标签：#rag

TL;DR Many LLM observability setups capture prompts, outputs, tokens, and latency while leaving retrieval failures hidden. A single search call may conceal query rewriting, filtering, fetching, deduplication, reranking, and evidence selection. A useful trace connects the original question to the effective query, returned sources, selected passages, and final claims. Retrieval tracing helps distinguish missing, stale, or ignored evidence from a genuine generation failure. Production teams should measure freshness, duplicate evidence, citation coverage, and cost per grounded answer. A user asks your AI assistant whether a product still supports a particular feature. The assistant responds confidently and links to the company’s documentation. The model request succeeded. Latency was normal. Token usage stayed within budget. No tool call failed. Every indicator on the dashboard is green. The answer is also six months out of date. The model trace cannot tell you whether the system searched for the wrong phrase, preferred an old page, discarded a better result, or ignored the correct evidence. It only shows the context that eventually reached the model. That is the blind spot in model-centred observability. For RAG applications and web-connected agents, the useful unit of observation is not the model call. It is the complete evidence path. A Successful Model Call Can Still Be a Failed Request A typical LLM trace records the prompt, response, model name, token consumption, latency, errors, and perhaps a tool invocation. That is useful for diagnosing slow requests, malformed inputs, and unexpectedly expensive generations. It does not tell you whether the model received the right facts. In a retrieval application, the final prompt is assembled by an upstream system. That system may rewrite the query, choose a search provider, apply time or domain filters, fetch pages, extract text, remove duplicates, rerank candidates, and select passages for the context window. The model ca

Status of the demo. The viewer was regenerated on August 29, 2026 from a sanitized public projection (with the non-public bits stripped out 😀): the artifact contains no mail bodies, attachments, addresses, absolute paths, tokens, credentials, or microdata. Preamble: remembering is not enough In the second part I went after a bounded problem: getting penta-agent 's memory to retrieve evidence and to recognize when it had found none. The question in this third part is more practical, and it comes out of the system having been in use for a while: what happens to a memory as it grows and turns blurry, or even contradictory? An index can pile up fragments without any trouble, and there are plenty of tools that already do that well. A more useful memory, in my judgment, has to carry provenance, currency, permissions, contradictions, and deletion criteria. It also has to tell finding a source apart from using it correctly. Recent literature insists on separating RAG — retrieval-augmented generation — context management, and agent memory, because they do different jobs and call for different evaluations 1 . What follows has three movements: what changed since part II; which experiments survived a more serious evaluation; and how to show a memory without passing it off as a mind. Spoonful 1: from retrieving fragments to governing evidence In part II the problem was retrieving well : finding the relevant context and recognizing when there was not enough evidence. A useful memory does not only retrieve information; it also has to know where it came from, whether it still holds, where it can be used, and what is allowed to be done with it . RAG mostly solves retrieval. The memory layer adds rules for keeping, updating, relating, or discarding evidence. None of those functions amounts, on its own, to identity. To describe provenance I use concepts compatible with PROV-O — entities, activities, and agents — while currency, sensitivity, and permissions need rules of their own 2 .

In the first post I described how I organized my local multi-agent setup, penta-agent : Codex executes, Claude reviews, other agents enter in bounded ways, and the human keeps closure authority. I also argued that operational memory should not depend on a single conversation or be confused with the vector index. By the time I closed that first post, I already had continuity mechanisms: handoffs, routing rules, append-only logs, experiential memory in JSONL/YAML, a rebuildable vector collection, and the recall-context skill. My problem was not absolute amnesia. It was that I still could not prove what the system retrieved, when it confused a coincidence with evidence, and when it should admit that it did not have an answer. This second part, then, is not about inventing memory from scratch. It is about turning still-fragile operational continuity into a traceable, testable, and rebuildable mechanism. The idea of an external working memory is not new. It echoes Bush's old ambition of augmenting recall through a personal archive and the extended-mind intuition that notes and tools can become part of cognition. 1 2 My claim here is narrower: local traces are useful only if I can retrieve them with provenance and audit how they were used. Spoonful 1: the problem was not storing, but retrieving well Storing information is easy. The difficult part, I think, is retrieving the right piece when there are successive decisions, similar names, contradictory versions, and explanations spread across several files. To organize that "memory" in my own setup, I separated its operational layers: Table 1 - System memory layers Layer Question it answers Effective implementation Canonical record What happened, and what was decided? memory/experience-events.jsonl , memory/experience-lessons.yaml , memory/interaction-metrics.jsonl , and curated context events. Retrieval index Where is the relevant evidence? Qdrant with penta_context_v2 for curated context and penta_experience_v1 for operat

Every RAG pipeline needs to convert text into vectors. The question is where that conversion happens. You have two options: run an embedding model locally on your own hardware, or call an API that runs the model on someone else's hardware. Both work. The right choice depends on your constraints — and understanding the tradeoffs is more useful than a recommendation. This article is about why I chose local embeddings with sentence-transformers/all-MiniLM-L6-v2 for this pipeline, and when I'd switch to an API. What Embeddings Actually Do Before the tradeoffs, a quick grounding on what's happening. An embedding model takes text and converts it into a fixed-size vector of floating-point numbers — a list of 384 numbers in the case of all-MiniLM-L6-v2 . That vector encodes the semantic meaning of the text in a way that allows mathematical comparison. Two pieces of text with similar meaning produce vectors that are close together in the 384-dimensional vector space. "Authentication failed" and "login was rejected" are semantically similar — their vectors will be close. "Authentication failed" and "quarterly revenue report" are semantically distant — their vectors will be far apart. This is what makes retrieval work. When you embed a query and search for the nearest chunks, you're finding chunks that are semantically similar to the question — not just chunks that contain the same keywords. The embedding model determines the quality of this semantic matching. A better model produces vectors where semantic similarity maps more accurately to vector proximity. The Local Embedding Choice My pipeline uses sentence-transformers/all-MiniLM-L6-v2 via ChromaDB's SentenceTransformerEmbeddingFunction : from chromadb.utils.embedding_functions import SentenceTransformerEmbeddingFunction embedding_fn = SentenceTransformerEmbeddingFunction ( model_name = " sentence-transformers/all-MiniLM-L6-v2 " ) This runs entirely on your local CPU. No API key, no network request, no cost per embedding,

The first version of a RAG system always looks convincing. You connect a document loader, a vector database, and a large model, ask a question, and the answer comes back with impressive confidence. Then production happens. A support agent asks about a refund policy that changed last week, and the bot answers with the old policy. A user from the finance team sees chunks they should never see. Gemini starts returning 429 errors during a reindex. A 3,000-document ingestion workflow fails at document 2,412, and you have no idea how to resume safely. That is the gap between a RAG demo and a production RAG pipeline. This walkthrough focuses on building a maintainable retrieval-augmented generation pipeline using n8n for orchestration, Qdrant for vector storage and filtered retrieval, and Gemini for embedding and answer generation. The goal is not just “make it answer.” The goal is to make it operable: idempotent ingestion, access-controlled retrieval, retry-safe automation, grounded answers, and a path for evaluation. TL;DR Treat RAG as two separate pipelines : ingestion and query. Store more than vectors in Qdrant: source_id , acl , version , updated_at , chunk_index , and text. Make ingestion idempotent so reprocessing documents does not create duplicate truth. Use Qdrant filters for permissions, freshness, and document status. Force Gemini to answer only from retrieved evidence and return citations. Add retries, timeouts, dead-letter handling, and evaluation before users do the testing for you. 📋 Table of Contents The Production Problem with Demo RAG 1. Split RAG Into Two Pipelines Before You Automate Anything 2. Design the Qdrant Collection Around Access Control and Freshness 3. Chunk for Retrieval, Not for Reading 4. Make Ingestion Idempotent and Resumable 5. Embed in Controlled Batches Without Dropping Documents 6. Retrieve With Filters, Not Blind Similarity 7. Make Gemini Prove It Used the Evidence 8. Add the Production Guardrails: Retries, Timeouts, and Dead Lette

A browser and a language model can look at the same URL and effectively see two different things. A browser sees a rendered interface: navigation, cookie banners, buttons, ads, sidebars, images, scripts, interactive components, and eventually the text a human came to read. A language model sees whatever representation we decide to give it. That distinction matters when the URL is going into a RAG pipeline. Open the developer tools on any major news or documentation site and look at the raw HTML. A typical article page runs between 300KB and 800KB of markup. The article text itself is usually between 2KB and 10KB. The ratio of markup to content is consistently between 10:1 and 40:1 depending on how heavily templated the site is. When you pass raw HTML to a language model, you are passing all of it, and most pipelines treat this as an acceptable default. Perceive is the endpoint we built to fix that. You give it a URL. It returns clean Markdown. This post is about what happens in between and why we made the engineering decisions we did. Why raw HTML is a poor RAG input The token waste is real but it is not the worst problem. Three failure modes compound each other. Token waste . A blog post with 800 words of real content can run to 6,000–12,000 tokens as raw HTML once you include navigation, scripts, inline styles, and layout markup. The same content in Markdown is often 900–1,200 tokens. That is not just a cost issue. It is context window space that cannot go to content. Embedding contamination . Embedding models are trained predominantly on natural language. When you embed a chunk containing <div class="sidebar-widget__title">Related Articles</div> alongside the article content, the vector is pulled toward the markup semantics rather than the content semantics . The embedding does not cleanly represent the article; it represents a mixture of the article and the site's component naming conventions. Retrieval degrades as a result: chunks that should be semantically si

How to Get YouTube Transcripts as a Developer (4 Methods That Work in 2026) YouTube transcripts unlock a lot: AI video summarizers, searchable course databases, RAG over video libraries, dataset generation for fine-tuning, repurposing videos into articles. But getting transcripts programmatically is full of sharp edges: disabled captions, rate limits, datacenter IP blocks, and YouTube's ever-changing frontend. This guide walks through every practical method with working code. Method 4 is the managed service I run. Skip ahead if you just want the API call. The DIY methods below are real and will serve you well for small jobs. What you're actually fetching YouTube stores captions as timed tracks in two flavors: Manual captions : uploaded by creators, best accuracy Auto-generated captions : YouTube's speech recognition, most videos Each track is text plus timing ( text/start/duration ), servable as SRT, VTT, or YouTube's timedtext XML. Everything below ultimately resolves to that shape. Method 1: youtube-transcript-api (Python) The standard open-source library. Start here for scripts and prototypes. pip install youtube-transcript-api from youtube_transcript_api import YouTubeTranscriptApi video_id = " dQw4w9WgXcQ " # the ID from the watch URL transcript = YouTubeTranscriptApi . get_transcript ( video_id ) for entry in transcript : print ( f " [ { entry [ ' start ' ] : . 2 f } s] { entry [ ' text ' ] } " ) It returns a list of dicts, one {'text', 'start', 'duration'} per segment. For other languages, list what's available first, then fetch or translate: tl = YouTubeTranscriptApi . list_transcripts ( video_id ) for t in tl : print ( t . language_code , " generated: " , t . is_generated ) transcript = YouTubeTranscriptApi . get_transcript ( video_id , languages = [ " id " , " en " ]) track = tl . find_transcript ([ " en " ]) translated = track . translate ( " id " ). fetch () # free, server-side Handle the caption-less case explicitly instead of catching bare Exception .

Adding citations to an AI answer feels like the moment the system becomes trustworthy. The response looks researched. Source links appear beside the text. The model is no longer answering only from its training data. But a cited answer can still be wrong. A citation may support a nearby sentence rather than the claim the user cares about. A source may be authoritative while the retrieved passage is stale. File Search may query the wrong store or document version. The model may retrieve good evidence and then write a conclusion that goes beyond it. Grounding is a capability. Trust still requires an application contract. Series note: This is Part 6 of Reliable Google AI Agents in TypeScript . The Interactions API examples use its post-May-2026 steps schema and were checked against @google/genai 2.21.0. The API remains beta, so pin and retest the SDK before copying production code. Retrieval success is not answer success Gemini can ground responses with Google Search for current public information and File Search for indexed domain-specific documents. The Interactions API exposes the execution steps and inline citation annotations, giving the application more evidence than a text completion alone. A minimal Google Search interaction looks like this: import { GoogleGenAI } from " @google/genai " ; const ai = new GoogleGenAI ({}); const interaction = await ai . interactions . create ({ model : process . env . GEMINI_MODEL ?? " gemini-3.8-flash " , input : " What changed in the public policy this week? " , tools : [{ type : " google_search " }], }); The synthesized text is only one part of the result. The steps show whether search occurred and where citations attach. type Citation = { title ?: string ; url ?: string ; citedText : string ; }; const citations : Citation [] = []; for ( const step of interaction . steps ?? []) { if ( step . type !== " model_output " ) continue ; for ( const contentBlock of step . content ?? []) { if ( contentBlock . type !== " text " ) contin

RAG Retrieval Gotchas at Scale: Insights and Solutions Retrieval-Augmented Generation (RAG) has emerged as a powerful paradigm in natural language processing (NLP), combining retrieval and generation to produce contextually relevant outputs. However, implementing RAG at scale introduces several challenges, or "gotchas," that can significantly impact performance and usability. In this article, we'll explore these pitfalls and provide concrete solutions, complete with code snippets and specific version numbers, to help you scale your RAG implementations effectively. Understanding RAG Architecture Before diving into the gotchas, it's essential to understand the architecture of RAG. The RAG model typically consists of two components: Retriever : This component fetches relevant documents from a large corpus based on a given query. Generator : This component generates a response based on the retrieved documents. In a typical RAG setup, you might use models from Hugging Face's Transformers library (version 4.21.1 or later is recommended) for both the retriever and generator. For instance, the RAG model can be set up as follows: from transformers import RagTokenizer , RagRetriever , RagSequenceForGeneration tokenizer = RagTokenizer . from_pretrained ( " facebook/rag-sequence-large " ) retriever = RagRetriever . from_pretrained ( " facebook/rag-sequence-large " ) model = RagSequenceForGeneration . from_pretrained ( " facebook/rag-sequence-large " ) Gotcha 1: Document Retrieval Latency Problem When scaling RAG systems, one common issue is the latency during document retrieval. If the retriever is querying a large corpus, the response time can significantly slow down the overall processing speed. Solution To mitigate this, consider optimizing your retrieval strategy. One approach is to use approximate nearest neighbor (ANN) search algorithms, such as FAISS (version 1.7.1), which can drastically reduce retrieval times. Here's a brief example of how to implement FAISS with your

I was up at 2 AM, staring at seventeen PDFs that refused to tell me anything. My midterm was in six days, and my notes were a mess of arrows, acronyms, and half-typed definitions. I wanted a chatbot that could answer questions about my own lectures. Not a fancy one. Just something that would take a question, find the relevant slide, and answer in plain language. So I built one. I used MonkeyCode for the free model access and free server space. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Their open-source platform's free tier includes 10 million tokens and a server slot, which is enough for a weekend prototype. The “why not” won. The plan was simple: extract text from the PDFs, split it into chunks, retrieve the most relevant chunks with a dumb similarity search, then ask a model to answer from those chunks. No vector database. No fine-tuning. Just a few lines of Python and a POST request. The extraction step was almost too easy. from pypdf import PdfReader def extract_pdf ( path ): return " \n " . join ( page . extract_text () for page in PdfReader ( path ). pages ) Most of my slides were text-heavy, so it worked. One deck came out as garbage because the pages were rotated. That was my first warning: garbage in, confident nonsense out. Next, chunking. I set a chunk size of 1,200 characters with an overlap of a hundred. Small enough to be relevant, big enough to contain a complete idea. def chunk_text ( text , size = 1200 , overlap = 100 ): chunks = [] for i in range ( 0 , len ( text ), size - overlap ): chunks . append ( text [ i : i + size ]) return chunks I didn't use a vector database. My whole corpus was about two hundred chunks, so TF-IDF plus cosine similarity was enough. More importantly, it made every retrieval transparent. I could see exactly which chunks the bot pulled, and why. from sklearn.feature_extraction.text import TfidfVectorizer from sklearn.metrics.pairwise import cosine_similarity def retrieve ( query , chunks

RAG Evaluation: How to Know if Your RAG System Actually Works You built a RAG chatbot. It answers questions from your documents. You test it a few times. The answers look good. So… can you ship it? No. One good answer doesn't tell you whether your RAG system works. A RAG application has multiple moving parts. The retriever can fail. The generator can fail. They can both work individually and still fail when combined. And once the application goes live, your users will ask questions you never tested. So how do you actually evaluate a RAG system? The answer is an eval suite . Components → Pipeline → Application → Regression → Online Evaluation This article walks through the same framework I use in my RAG evaluation video. ▶ Watch the full video The Problem: “It Feels Better” Isn't an Evaluation Imagine you're building an airline support chatbot for a fictional airline called SkyHigh Airlines . Passengers can ask questions about: Baggage Refunds Pets Travel policies The chatbot uses RAG to search the airline's policy documents and generate an answer. A passenger asks: “How much does it cost to bring my cat?” The chatbot responds: “Bringing your cat costs $95.” Looks good. But what if the retriever found the wrong document and the model happened to generate something plausible? Or what if the retriever found the correct policy, but the model ignored it and invented the answer? From the outside, both problems look identical: Bad answer. But they require completely different fixes. That's why you can't evaluate RAG as one giant black box. You need to test the pieces separately. First: Build a Golden Set Before measuring anything, you need something to measure against. Create a fixed set of questions that represent the kinds of questions your users will actually ask. For our SkyHigh chatbot, imagine we create 50 questions about the airline's policies. For every question, we record: The question The correct answer The document chunks that should contain the answer For examp

You've probably seen the term RAG everywhere lately — "RAG pipeline," "RAG chatbot," "build your own RAG app." It sounds complicated, but the idea behind it is actually pretty simple. In this article, I'll explain RAG in plain language, then walk through how it works using a real project I built: Guidely , an internal knowledge assistant that answers questions using a company's own documents. The Problem RAG Solves Large language models (like GPT or Claude) are trained on a huge amount of general knowledge, but they don't know about your specific data — your company's internal docs, your product manuals, your onboarding guides. They also can't be retrained every time a document changes; that's slow and expensive. RAG solves this without retraining the model at all. Basically: RAG means: before answering a question, first go find the relevant pieces of your own documents, and hand those to the AI along with the question. That's it. "Retrieval" (finding the right information) + "Augmented Generation" (the AI answers using that information). Instead of the AI answering from memory alone, it answers using facts you hand it in the moment. The Three Core Pieces Let's break down the three things you need to make this work: chunking , embeddings , and vector search . 1. Chunking — Breaking Documents Into Pieces You can't hand an AI model an entire 200-page document and ask it to search through it efficiently. So the first step is splitting documents into smaller, manageable pieces called chunks . In Guidely, I used a token-window chunker — it splits text based on a fixed number of tokens (roughly, pieces of words) per chunk, rather than just splitting by paragraph or sentence. This matters because: Chunks that are too big waste space and slow things down. Chunks that are too small lose context and produce confusing answers. A token-window approach gives you consistent, predictable chunk sizes, which makes the next steps more reliable. 2. Embeddings — Turning Text Into Numbe

Intro I recently read an article about a VC who uses AI to boost his productivity. He described building a knowledge base using NotebookLM, and one point that stuck with me was: Every time I read something online that I thought I wanted to remember, I'd copy and paste it into that repository. Whenever I wanted to write a blog post, I could query it and retrieve all the information I needed. Like him, I have knowledge and resources scattered across Logseq, Gmail, Notion, ADR documents, Slack, project readmes, Markdown files, Twitter, and more. That made me wonder: how could I build my own system? Tools like NotebookLM exist, but I want a single knowledge layer across all my sources—not isolated, manually managed workspaces. NotebookLM’s model requires creating a workspace, adding sources, and asking questions about them, but separate notebooks mean separate contexts. As an experienced engineer who’s never built a Retrieval-Augmented Generation (RAG) system, I saw this as an opportunity to learn and share. I’ll approach it from first principles, and in this series, we’ll: Architect a RAG system from the ground up. Break its subsystems down and clarify their responsibilities. Identify architectural decisions and tradeoffs. Integrate the RAG system with an LLM to create something like a personal Google Search for your whole digital life. Use Case Two years ago, I read an article about a man with ADHD. The post stayed with me, but for over a year I couldn’t find it again, even after searching bookmarks and Googling "article about a guy with ADHD". I finally found it because the author emailed it to his mailing list. Without that email, I might never have seen it again. With a personal knowledge base (RAG system), I could have simply asked for "an article about a guy with ADHD" and quickly found it. Let’s dive into how such a system works. What is Retrieval-Augmented Generation (RAG)? Retrieval-Augmented Generation is the process of supplementing LLM (Large Language Model

The assumption every RAG demo makes Standard RAG assumes the user's question maps onto one vector search. One query in, one embedding, one top-k lookup, one answer. That assumption holds up in demos, because demos ask demo questions. "What's our parental leave policy?" is one document. Retrieve it, stuff it into the prompt, done. Then you ship, and a real user types: "Did the carrier rate change we approved in Q2 actually reduce our cost per shipment in the Northeast, and does that hold if I exclude the Boston depot?" That question needs a policy document, a rate table, a transactional aggregate, and a filtered re-computation. Your retriever will embed the whole sentence, find the three chunks nearest to it in vector space, and hand the model text that is topically adjacent and factually useless. The model, being a good sport, will answer anyway. The problem isn't the embedding model or the chunk size. You hardcoded how many times to retrieve, and where to retrieve from, at design time, for a question you hadn't read yet. Agentic RAG moves that decision to runtime. Planners, memory, MCP servers, sub-agents: all of it is implementation detail hanging off that one change. Architecture 1: standard RAG is a straight line STANDARD RAG — fixed pipeline, one pass ┌──────┐ 1. prompt+query ┌─────────────┐ │ User │ ───────────────────► │ Chat UI │ └──────┘ └──────┬──────┘ ▲ │ 2. query │ 6. response ▼ │ ┌─────────────┐ │ │ Retriever │ │ └──────┬──────┘ │ │ 3. fetch (top-k, one shot) │ ▼ │ ┌───────────────────────────┐ │ │ Knowledge Sources │ │ │ docs · PDFs · code · DB │ │ │ APIs · web index │ │ └───────────┬───────────────┘ │ │ 4. chunks │ ┌──────▼──────┐ └──────────────────────────│ LLM │ └─────────────┘ 5. prompt + query + enhanced context The defining property is that the model is never consulted about retrieval. It receives context and produces text, and retrieval already finished by the time it runs. That's a design choice with real advantages. One embedding call plus on

As part of the Gen AI Academy APAC , I set out to solve a major pain point for educators: manually sifting through textbooks to create grade-appropriate question papers. I built an automated Question Paper Generator using a Serverless Next.js stack, a Retrieval-Augmented Generation (RAG) architecture, and the complete Google Cloud AI suite. Teachers simply upload a textbook chapter (PDF), specify the grade and subject, and let the AI generate a fully formatted assessment quiz. While the architecture sounds straightforward, orchestrating these enterprise-grade APIs in a serverless environment presented several intense technical hurdles. Here is a deep dive into the architecture, the specific roadblocks I hit, and how I ultimately solved them. 🏗️ The RAG Architecture The application is built on Next.js 15 and deployed to Google Cloud Run . The pipeline flows as follows: Document Extraction : The PDF is uploaded and sent to Google Cloud Document AI (Document OCR Processor) to extract the raw text. Chunking & Embeddings : The text is chunked into logical paragraphs and sent to Vertex AI ( text-embedding-004 ) to generate dense vector embeddings. Vector Database : The embeddings and metadata (Grade, Subject) are stored seamlessly in Firestore using native VectorValue support. Retrieval & Generation : When a teacher requests a quiz, the query is embedded, and a findNearest Vector Search runs on Firestore. The retrieved context is passed to Google Gen AI ( gemini-3.5-flash ) to synthesize the structured question paper. 🐛 The Technical Challenges & How I Solved Them Building an end-to-end pipeline using cutting-edge SDKs often means dealing with strict schema validations and opaque error codes. Here are the major technical gotchas I faced. 1. The Document AI Region Endpoint Mismatch The Challenge: I provisioned a Document OCR processor in the asia-south1 region. However, when my Node.js client attempted to send a processing request using the processor's full resource name,

As part of my AI Engineering journey, I recently worked on a project that helped me understand how Retrieval-Augmented Generation (RAG) works in practice. I built a Hybrid RAG system that combines FAISS vector search and BM25 keyword search to retrieve relevant information from a knowledge base and use it to generate grounded answers. In this post, I’ll briefly share what I built, how the system works, and some of the things I learned along the way. Why RAG? Large Language Models are great at generating natural-language responses, but they may not have access to information contained in a specific document or knowledge base. RAG addresses this by first retrieving relevant information from an external knowledge base and then providing that information to the LLM as context. The basic workflow is: User Query ↓ Retrieve Relevant Information ↓ Provide Context to LLM ↓ Generate Answer For my project, I wanted to take this a step further by combining semantic search and keyword search. 🔍 Hybrid Retrieval The system uses two retrieval methods: Vector Search with FAISS Document content is divided into smaller chunks and converted into vector embeddings. These embeddings are stored in a FAISS index, which is used to find documents that are semantically similar to the user’s query. This is useful even when the query and the document use different wording. Keyword Search with BM25 The second retrieval method is BM25. BM25 focuses on the occurrence and importance of terms in the query and documents. This makes it useful for exact terminology, technical terms, names, and identifiers. Instead of depending on only one retrieval method, both approaches are combined. User Query │ ┌──────────┴──────────┐ ↓ ↓ FAISS Search BM25 Search Semantic Search Keyword Search │ │ └──────────┬──────────┘ ↓ Hybrid Ranking ↓ Relevant Context ↓ LLM ↓ Final Answer The FAISS and BM25 scores are normalized and combined using weighted scoring. The results are then ranked, and the highest-ranked chunks ar

In my first post, I wrote about why I spent my first week writing zero business logic and instead built rig - our lightweight, POSIX-compliant local provisioning tool. It was my way of rejecting "wiki-ops" and applying Infrastructure-as-Code (IaC) discipline to our local environments so that a hardware failure means minutes of downtime, not a week. But as I transitioned into Week Two, I was hit by a different kind of operational reality check. For years, I had been building a comprehensive repository of system architecture, design decisions, and guidelines on Confluence. It was my digital home. So, knowing I would be creating a startup, I set to work writing my documentation in my spare time in preparation. But during a brief hiatus of inactivity, the space was silently, unceremoniously deleted. It was gone. Late nights of ideas, patterns, templates, and reference materials vanished into the cloud ether. That loss was a violent reminder of a lesson I thought I'd fully mastered: if your documentation doesn't live alongside your code, you don't truly own it. Relying on third-party SaaS wikis to store the soul of your system architecture is just another form of "click-ops". It creates an artificial separation between the craftsmen writing the logic and the documentation that defines it. But rather than mourning my lost Confluence space, I treated it as a catalyst. I decided that our young startup would not have a bloated, detached corporate wiki. Instead, we would treat Documentation as a Contract - a unified, git-backed human-and-machine contract that serves as the precise, zero-maintenance boundary for our AI systems. Here is how losing my documentation led to a new architectural philosophy, and how we built a zero-overhead, "Anti-AI AI Strategy" that uses GitLab CI/CD and Google Workspace to run a secure, managed RAG pipeline. The Anti-AI Strategy: Why We Refuse to Let AI Write Our Code Walk into almost any tech startup today, and you’ll find developers blindly feed

You self-host S3-compatible storage on bare metal by installing a single Rust binary on a Linux server and pointing any S3 client at it. RustFS installs with one script, listens on port 9000 (S3 API) and 9001 (console), and is Apache 2.0 licensed. Single-node mode is production-ready today; multi-node clustering is still under testing. Every command below is copied verbatim from the official source cited beside it. This sandbox has no Docker daemon, so none of the commands were executed here; they are marked accordingly. Key Stats Fact Source RustFS installs with one command and runs as a systemd service on x86_64 or aarch64 Linux RustFS docs (Linux quick-start) Default S3 API port is 9000; console port is 9001 RustFS GitHub README Default credentials are rustfsadmin / rustfsadmin and must be changed RustFS README + docs RustFS is Apache 2.0 licensed and S3-compatible RustFS GitHub README Single-node mode is production-ready; distributed mode is still under testing RustFS README Feature & Status What is self-hosted S3-compatible storage? A self-hosted S3-compatible storage server is a program you run on your own hardware that speaks the Amazon S3 API. Applications using AWS SDKs, the aws CLI, or MinIO's mc can talk to it without code changes, because the bucket, object, and credential model matches S3. The difference from a cloud bucket is ownership: the disks, the network path, and the uptime are yours. RustFS is one such server, written in Rust and licensed under Apache 2.0. It exposes the S3 API on port 9000 and a web console on 9001, and it stores objects on the local filesystem. Because it is S3-compatible, the same client code that targets AWS S3 also targets a RustFS node. That compatibility is the whole point of self-hosting here: you get an S3 endpoint without renting one. Why run object storage on bare metal? Running object storage on bare metal means installing the server directly on a Linux machine instead of in a container or a managed cloud. The appeal

Most LlamaIndex setups end up with two separate backends once you go beyond plain vector search: a vector store for VectorStoreIndex , and a separate graph database for PropertyGraphIndex when you need relationship-aware retrieval (GraphRAG). Two services, two connection strings, two things to keep in sync. This is a walkthrough of backing both index types with SynapCores instead — one engine, one connection, both index types. Setup docker run -d --name synapcores -p 8080:8080 \ -e AIDB_ACCEPT_LICENSE = 1 \ -v synapcores-data:/var/lib/synapcores \ ghcr.io/synapcores/community:latest pip install llama-index llama-index-vector-stores-synapcores llama-index-graph-stores-synapcores Both integration packages are independently published on PyPI: llama-index-vector-stores-synapcores llama-index-graph-stores-synapcores Vector store — standard RAG from llama_index.core import VectorStoreIndex , StorageContext , Document from llama_index.vector_stores.synapcores import SynapCoresVectorStore vector_store = SynapCoresVectorStore ( uri = " http://localhost:8080 " , embedding_dim = 1536 ) storage_context = StorageContext . from_defaults ( vector_store = vector_store ) docs = [ Document ( text = " SynapCores runs vector search, graph traversal, and SQL in one engine. " )] index = VectorStoreIndex . from_documents ( docs , storage_context = storage_context ) query_engine = index . as_query_engine () response = query_engine . query ( " What does SynapCores combine into one engine? " ) print ( response ) The vector store implements the full BasePydanticVectorStore ABC — add , delete , query , delete_nodes , clear , plus the async surface. Metadata filtering supports the full MetadataFilters grammar: all 12 operators ( EQ , NE , GT / GTE / LT / LTE , IN , NIN , TEXT_MATCH , TEXT_MATCH_INSENSITIVE , CONTAINS , IS_EMPTY ) with AND / OR / NOT and nested groups — so you're not giving up filtering power by moving off a dedicated vector DB. If you already have data in SynapCores from a prev

A practical checklist for confirming logs are collected, routed, stored, and reviewable Logs are useful only if they are available when the team needs them. In OCI, it is possible to enable service logs, route them through Connector Hub, and store them in Object Storage for later review. Connector Hub is also referenced in some Oracle material as Service Connector Hub. The setup can look simple on the surface. But from a delivery point of view, the important question is not whether the connector was created. The important question is: Can we prove that the logs are being collected, routed, stored, retained, and reviewed when needed? This article is written from a practical validation point of view. It uses a simple example: moving OCI Load Balancer logs from OCI Logging to Object Storage using Connector Hub. Scope note: this is an independent review and validation exercise. It is not a client implementation, and no production environment, customer data, or confidential information is referenced. All names, prefixes, and identifiers below are placeholders. Console labels, defaults, and behaviour can change between releases and regions, so every value should be confirmed in your own tenancy and current Oracle documentation. The goal is not to describe every possible logging design. The goal is to give a clear checklist that helps confirm the flow is working end to end. Why log retention needs validation Enabling a log is not the same as retaining a log. A team may be able to show that logging was switched on. That does not automatically prove that the data still exists for the period being questioned, that it landed where it was supposed to land, or that someone can retrieve and read it when needed. There is one detail worth stating early. There are two retention clocks, not one. Clock What it controls Where it is set Logging retention How long the log data stays inside OCI Logging On the individual log Object Storage lifecycle How long the exported copy stays in the