标签：#Agents

Optimizing RAG at Scale: Chunking, Retrieval, and the Bayesian Search That Cut Latency 40% How we moved from "semantic search + hope" to a measured, tunable retrieval pipeline with 95% recall@10 The RAG Reality Check Everyone ships RAG the same way: chunk by 512 tokens, embed with text-embedding-3-small , top-k=5, stuff into context. It works for demos. Then you hit production: Legal contracts: 512 tokens splits clauses mid-sentence API docs: 1000-token chunks drown signal in noise Customer tickets: Conversational context needs overlap, not fixed windows Latency: 500ms embedding + 200ms vector search + 300ms LLM = 1s+ per query We rebuilt our retrieval layer from first principles. Here's what actually moves metrics. Chunking: One Size Fits None # rag/chunking.py from abc import ABC , abstractmethod from dataclasses import dataclass @dataclass class Chunk : text : str metadata : dict token_count : int chunk_id : str class ChunkingStrategy ( ABC ): @abstractmethod def chunk ( self , document : str , metadata : dict ) -> list [ Chunk ]: ... class FixedTokenChunker ( ChunkingStrategy ): """ Baseline. Good for homogeneous content. """ def __init__ ( self , chunk_size = 512 , overlap = 50 ): self . chunk_size = chunk_size self . overlap = overlap class RecursiveChunker ( ChunkingStrategy ): """ Respects structure: markdown headers, code blocks, paragraphs. """ def __init__ ( self , separators = [ " \n ## " , " \n ### " , " \n\n " , " \n " , " " ], chunk_size = 512 ): self . separators = separators self . chunk_size = chunk_size class SemanticChunker ( ChunkingStrategy ): """ Uses embedding similarity to find natural boundaries. """ def __init__ ( self , model = " text-embedding-3-small " , threshold = 0.7 ): self . model = model self . threshold = threshold class AgenticChunker ( ChunkingStrategy ): """ LLM decides boundaries. Expensive but highest quality for complex docs. """ def __init__ ( self , model = " gpt-4o-mini " ): self . model = model Our production config by

Building Production-Grade LLM Evaluation Pipelines: From Vibes to Metrics How we replaced "looks good to me" with automated evaluation catching 92% of hallucinations before deployment The Problem: Why "Vibe Checks" Fail in Production Three months ago, our team shipped a RAG-based customer support assistant. It worked great in testing — we'd ask it questions, read the answers, and say "yeah, that looks right." Then it hit production. A customer asked about their billing cycle. The assistant confidently cited a policy that didn't exist. Another asked about API rate limits and got numbers from a competitor's documentation. By the time we caught it, 500+ users had seen hallucinated responses. The post-mortem was brutal: we had zero automated evaluation . Our test process was literally "ask 5 questions, read answers, thumbs up." What Production Evaluation Actually Needs Academic benchmarks (MMLU, HellaSwag) don't tell you if your system works for your use case. Production evaluation needs: Domain-specific judges — Your criteria, not generic "helpfulness" Speed — Evaluation must run in CI/CD, not overnight Regression detection — Know immediately when a prompt change breaks things CI/CD integration — Block merges that degrade quality Golden dataset management — Versioned, stratified, growing test cases Architecture: The Evaluation Pipeline ┌─────────────┐ ┌──────────────┐ ┌────────────────────┐ ┌──────────────┐ │ Test Cases │────▶│ LLM Under │────▶│ Judge Ensemble │────▶│ Metrics & │ │ (Golden Set)│ │ Test │ │ - Faithfulness │ │ Regression │ └─────────────┘ └──────────────┘ │ - Instruction F. │ │ Detection │ │ - JSON Schema │ └──────┬───────┘ │ - Custom LLM │ ▼ └────────────────────┘ ┌──────────────┐ │ Dashboard/ │ │ PR Comments │ └──────────────┘ Core Abstractions # eval/base.py @dataclass ( frozen = True ) class TestCase : id : str input : dict [ str , Any ] expected : dict [ str , Any ] | None = None tags : list [ str ] = field ( default_factory = list ) # ["edge-case",

Optimizing RAG at Scale: Chunking, Retrieval, and the Bayesian Search That Cut Latency 40% How we moved from "semantic search + hope" to a measured, tunable retrieval pipeline with 95% recall@10 The RAG Reality Check Everyone ships RAG the same way: chunk by 512 tokens, embed with text-embedding-3-small , top-k=5, stuff into context. It works for demos. Then you hit production: Legal contracts: 512 tokens splits clauses mid-sentence API docs: 1000-token chunks drown signal in noise Customer tickets: Conversational context needs overlap, not fixed windows Latency: 500ms embedding + 200ms vector search + 300ms LLM = 1s+ per query We rebuilt our retrieval layer from first principles. Here's what actually moves metrics. Chunking: One Size Fits None # rag/chunking.py from abc import ABC , abstractmethod from dataclasses import dataclass @dataclass class Chunk : text : str metadata : dict token_count : int chunk_id : str class ChunkingStrategy ( ABC ): @abstractmethod def chunk ( self , document : str , metadata : dict ) -> list [ Chunk ]: ... class FixedTokenChunker ( ChunkingStrategy ): """ Baseline. Good for homogeneous content. """ def __init__ ( self , chunk_size = 512 , overlap = 50 ): self . chunk_size = chunk_size self . overlap = overlap class RecursiveChunker ( ChunkingStrategy ): """ Respects structure: markdown headers, code blocks, paragraphs. """ def __init__ ( self , separators = [ " \n ## " , " \n ### " , " \n\n " , " \n " , " " ], chunk_size = 512 ): self . separators = separators self . chunk_size = chunk_size class SemanticChunker ( ChunkingStrategy ): """ Uses embedding similarity to find natural boundaries. """ def __init__ ( self , model = " text-embedding-3-small " , threshold = 0.7 ): self . model = model self . threshold = threshold class AgenticChunker ( ChunkingStrategy ): """ LLM decides boundaries. Expensive but highest quality for complex docs. """ def __init__ ( self , model = " gpt-4o-mini " ): self . model = model Our production config by

Building Production-Grade LLM Evaluation Pipelines: From Vibes to Metrics How we replaced "looks good to me" with automated evaluation catching 92% of hallucinations before deployment The Problem: Why "Vibe Checks" Fail in Production Three months ago, our team shipped a RAG-based customer support assistant. It worked great in testing — we'd ask it questions, read the answers, and say "yeah, that looks right." Then it hit production. A customer asked about their billing cycle. The assistant confidently cited a policy that didn't exist. Another asked about API rate limits and got numbers from a competitor's documentation. By the time we caught it, 500+ users had seen hallucinated responses. The post-mortem was brutal: we had zero automated evaluation . Our test process was literally "ask 5 questions, read answers, thumbs up." What Production Evaluation Actually Needs Academic benchmarks (MMLU, HellaSwag) don't tell you if your system works for your use case. Production evaluation needs: Domain-specific judges — Your criteria, not generic "helpfulness" Speed — Evaluation must run in CI/CD, not overnight Regression detection — Know immediately when a prompt change breaks things CI/CD integration — Block merges that degrade quality Golden dataset management — Versioned, stratified, growing test cases Architecture: The Evaluation Pipeline ┌─────────────┐ ┌──────────────┐ ┌────────────────────┐ ┌──────────────┐ │ Test Cases │────▶│ LLM Under │────▶│ Judge Ensemble │────▶│ Metrics & │ │ (Golden Set)│ │ Test │ │ - Faithfulness │ │ Regression │ └─────────────┘ └──────────────┘ │ - Instruction F. │ │ Detection │ │ - JSON Schema │ └──────┬───────┘ │ - Custom LLM │ ▼ └────────────────────┘ ┌──────────────┐ │ Dashboard/ │ │ PR Comments │ └──────────────┘ Core Abstractions # eval/base.py @dataclass ( frozen = True ) class TestCase : id : str input : dict [ str , Any ] expected : dict [ str , Any ] | None = None tags : list [ str ] = field ( default_factory = list ) # ["edge-case",

I've wanted to build this for about ten years, probably more like 15, and for most of those years it wasn't buildable. The idea never really changed: a world full of artificial beings, each with its own preferences, fears, personality, and instincts, dropped into a place with scarcity and weather and each other — just to see what they'd do, and how they'd treat one another. The thing that kept it on the shelf was always the same. The minds. If you want a world like that and you're working pre-LLM, you have two options. You script every reaction — finite state machines, behavior trees, utility AI — and you get a puppet show. It can be a good puppet show, but every interesting thing in it is something you wrote, which means it's not an experiment, it's an illustration. Or you train something bespoke, which for a solo developer with a side obsession was not happening. So the idea sat there for years, the way ideas do. Modern LLMs are the first thing that made the minds plausible. Not perfect — I'll be honest about the limits throughout this post — but plausible enough that an agent can reason about its own situation instead of executing my decision tree. So I went and built the world: Artificiety , a fantasy world that runs 24/7 and whose only inhabitants are AI agents. No human players inside it. You can watch it, you can put your own agent in, but you can't be a character in it. That constraint is the whole point. This post is about how it's built and what turned out to be hard. I'll try to keep the marketing to one link at the very end. What an agent actually is The cleanest way to think about an agent here: it's an LLM driving a character through a game API, nothing more. Once per tick, each agent runs a loop: Observe. It receives its local surroundings as structured data — what's near it, what's happening, the state of its own body and inventory, recent events. Not the whole world; just what that character could plausibly perceive. Decide. The model picks an actio

Optimizing RAG at Scale: Chunking, Retrieval, and the Bayesian Search That Cut Latency 40% How we moved from "semantic search + hope" to a measured, tunable retrieval pipeline with 95% recall@10 The RAG Reality Check Everyone ships RAG the same way: chunk by 512 tokens, embed with text-embedding-3-small , top-k=5, stuff into context. It works for demos. Then you hit production: Legal contracts: 512 tokens splits clauses mid-sentence API docs: 1000-token chunks drown signal in noise Customer tickets: Conversational context needs overlap, not fixed windows Latency: 500ms embedding + 200ms vector search + 300ms LLM = 1s+ per query We rebuilt our retrieval layer from first principles. Here's what actually moves metrics. Chunking: One Size Fits None # rag/chunking.py from abc import ABC , abstractmethod from dataclasses import dataclass @dataclass class Chunk : text : str metadata : dict token_count : int chunk_id : str class ChunkingStrategy ( ABC ): @abstractmethod def chunk ( self , document : str , metadata : dict ) -> list [ Chunk ]: ... class FixedTokenChunker ( ChunkingStrategy ): """ Baseline. Good for homogeneous content. """ def __init__ ( self , chunk_size = 512 , overlap = 50 ): self . chunk_size = chunk_size self . overlap = overlap class RecursiveChunker ( ChunkingStrategy ): """ Respects structure: markdown headers, code blocks, paragraphs. """ def __init__ ( self , separators = [ " \n ## " , " \n ### " , " \n\n " , " \n " , " " ], chunk_size = 512 ): self . separators = separators self . chunk_size = chunk_size class SemanticChunker ( ChunkingStrategy ): """ Uses embedding similarity to find natural boundaries. """ def __init__ ( self , model = " text-embedding-3-small " , threshold = 0.7 ): self . model = model self . threshold = threshold class AgenticChunker ( ChunkingStrategy ): """ LLM decides boundaries. Expensive but highest quality for complex docs. """ def __init__ ( self , model = " gpt-4o-mini " ): self . model = model Our production config by

Building Production-Grade LLM Evaluation Pipelines: From Vibes to Metrics How we replaced "looks good to me" with automated evaluation catching 92% of hallucinations before deployment The Problem: Why "Vibe Checks" Fail in Production Three months ago, our team shipped a RAG-based customer support assistant. It worked great in testing — we'd ask it questions, read the answers, and say "yeah, that looks right." Then it hit production. A customer asked about their billing cycle. The assistant confidently cited a policy that didn't exist. Another asked about API rate limits and got numbers from a competitor's documentation. By the time we caught it, 500+ users had seen hallucinated responses. The post-mortem was brutal: we had zero automated evaluation . Our test process was literally "ask 5 questions, read answers, thumbs up." What Production Evaluation Actually Needs Academic benchmarks (MMLU, HellaSwag) don't tell you if your system works for your use case. Production evaluation needs: Domain-specific judges — Your criteria, not generic "helpfulness" Speed — Evaluation must run in CI/CD, not overnight Regression detection — Know immediately when a prompt change breaks things CI/CD integration — Block merges that degrade quality Golden dataset management — Versioned, stratified, growing test cases Architecture: The Evaluation Pipeline ┌─────────────┐ ┌──────────────┐ ┌────────────────────┐ ┌──────────────┐ │ Test Cases │────▶│ LLM Under │────▶│ Judge Ensemble │────▶│ Metrics & │ │ (Golden Set)│ │ Test │ │ - Faithfulness │ │ Regression │ └─────────────┘ └──────────────┘ │ - Instruction F. │ │ Detection │ │ - JSON Schema │ └──────┬───────┘ │ - Custom LLM │ ▼ └────────────────────┘ ┌──────────────┐ │ Dashboard/ │ │ PR Comments │ └──────────────┘ Core Abstractions # eval/base.py @dataclass ( frozen = True ) class TestCase : id : str input : dict [ str , Any ] expected : dict [ str , Any ] | None = None tags : list [ str ] = field ( default_factory = list ) # ["edge-case",

When people demo "agents that automate business workflows," the demo usually ends right where the real problem begins: the moment the agent's output touches the real world. A wrong chatbot answer is annoying. A wrong official violation letter to a restaurant, sent under a county's letterhead, is a lawsuit. For the Qwen Cloud hackathon I built Inspection Autopilot, an agent that does the follow-up paperwork for a county food-safety office, on real public data: 1,333 inspections and 3,579 violation records from Clayton County, Georgia. The interesting part is not the agent. It's the governance around it, and the numbers that prove it works. Start with the receipts Before trusting an agent's judgment, test it against reality. We replayed the county's own history: 350 real (inspection, next-inspection) pairs, each triaged live by qwen-plus using only information available at the time. Facilities the agent flagged URGENT went on to fail their next real inspection 67.6% of the time. Facilities it cleared as ROUTINE failed only 21.1%, against a 48.6% base rate. The tiers are not vibes; the future agreed with them. Three design rules 1. Citations are verified in code, not vibes. The triage agent must justify every risk call by citing violation lines copied verbatim from the inspection record. The backend checks every citation against the source; anything that does not match is dropped and counted. On the committed live eval (50 inspections, 28 of them deliberately dangerous cases), the measured hallucination rate is 0.0% across 126 citations. And because a checker that never fires is indistinguishable from one that does not work, we sabotaged it: 50 forged citations injected, half invented outright, half real violation text lifted from different inspections, the forgery a lazy validator would miss. It caught 50 of 50 and preserved all 75 legitimate citations. We know the tripwire works because we set it off. 2. The log is append-only. Proposals and decisions are insert-only

This post is my submission for DEV Education Track: Build Multi-Agent Systems with ADK . What I Built Preparing for technical interviews can be overwhelming. I wanted to build a tool that doesn't just give generic questions, but actually analyzes my specific resume to challenge my unique skill set. This led me to build a multi-agent system using the ADK. This multi-agent system, will take user resume and extracts their profile for generating Interview Questions specialized to the candidate profile. The agents communicate in a sequential loop: The Profiler extracts data -> The Interviewer generates questions -> The Judge validates them. If the Judge rejects a question, the Interviewer re-drafts it, ensuring only high-quality, resume-relevant questions make it to the user. Cloud Run Embed Your Agents Profiler Receives a resume PDF GCS path, downloads it in-memory, parses the text content, and builds a summary of skills. Interviewer Reads the candidate summary and drafts 3 technical interview questions designed to test the boundaries of their experience. Analyzes the drafted questions. Passes the iteration if they are resume-specific; rejects/fails them if they are too generic. Key Learnings This project was a fantastic weekend challenge. Working through the Google Codelab gave me a solid grasp of agent-based architectures, specifically implementing Agent, LoopAgent, and SequentialAgent to create a robust workflow. A few key technical takeaways included: Managing Statelessness : Learning to handle agent sessions in a Cloud Run environment was a great lesson in explicit session lifecycle management. Cloud Integration : Integrating Google Cloud Storage for file handling taught me how to bridge in-memory document processing with persistent cloud storage efficiently. Deployment Architecture : Mastering the transition from local development to a containerized, production-ready Cloud Run deployment provided deep insights into modern backend orchestration. Check the Code from

Over the past few months, I've been exploring how AI can move beyond chatbots and become an active part of business workflows. That journey led me to build SmartStock AI , an inventory management platform that combines modern web technologies with AI agents, Retrieval-Augmented Generation (RAG), demand forecasting, and automation. Instead of only tracking inventory, SmartStock AI helps businesses make proactive decisions. What SmartStock AI Can Do 🤖 Forecast future product demand 📦 Recommend purchasing decisions through AI agents 📚 Answer inventory-related questions using Hybrid RAG with source citations 📄 Process invoices using multimodal AI ⚡ Generate real-time inventory alerts 🔐 Secure the platform with JWT authentication and role-based access control Technology Stack Frontend React 19 Backend Django 5 Django REST Framework Database PostgreSQL pgvector AI LangChain Hybrid RAG AI Agents Prophet Forecasting Infrastructure Celery Redis Docker GitHub Actions What I Learned Building SmartStock AI taught me much more than integrating an LLM into an application. Some of the biggest lessons were: Designing AI features that solve real business problems. Building reliable agent workflows instead of simple chatbot interactions. Combining vector search with structured database queries. Managing asynchronous AI tasks using Celery and Redis. Creating production-ready APIs with Django REST Framework. Deploying and maintaining a modern full-stack application. Demo 🎥 YouTube Demo https://www.youtube.com/watch?v=DQJqs6bgE98 🌐 Live Demo https://smart-stock-dev.vercel.app/ Demo Account Email: viewer@smartstock.ai Password: Viewer123! 💻 GitHub Repository https://github.com/Eng-Ayman-Mohamed/SmartStock-AI Final Thoughts This project was developed as my graduation project during the Information Technology Institute (ITI) Full Stack Web & Generative AI Program. It was an incredible opportunity to explore AI engineering, backend architecture, and modern software development while buildin

Reading a public website and changing something inside a signed-in account are not the same kind of job. I wanted an AI agent to collect the first 10 Hacker News stories, explain how it gathered them, and place that work inside my DEV editor. Save the article as a draft. Do not publish it. The hard part was not finding the stories. It was giving the agent enough access to finish inside my signed-in account without giving it permission to publish. That instruction contained two jobs with different access needs: Read public data from Hacker News. Change private account state by creating a draft in my signed-in DEV account. Using a full browser for the first job would add machinery and access that the result did not need. The public fetch path had neither the authenticated context nor the authority required for the second job. The useful question was not “Which browser tool should do everything?” It was “Where does this task actually cross into a browser?” Disclosure: I work on Unchained, SearchAgentSky, and Unbrowser. This article was prepared with AI assistance and reviewed before publication. The public step needed only fetch I initially opened Hacker News in a lightweight page session and queried its story links. It worked—but that did not mean it was necessary. Hacker News has a public API . The same result could be produced with ordinary HTTP: const base = " https://hacker-news.firebaseio.com/v0 " ; const ids = await fetch ( ` ${ base } /topstories.json` ). then ( r => r . json ()); const stories = await Promise . all ( ids . slice ( 0 , 10 ). map ( async ( id , index ) => { const story = await fetch ( ` ${ base } /item/ ${ id } .json` ). then ( r => r . json ()); return { rank : index + 1 , title : story . title , url : story . url ?? `https://news.ycombinator.com/item?id= ${ id } ` }; })); console . log ( JSON . stringify ( stories , null , 2 )); At 2026-07-19T03:19:04Z , the page query returned 30 Hacker News story anchors. At 2026-07-19T03:35:29.670Z , fetch

If you're a .NET developer looking to break into AI engineering, agents are the single best place to start. They're the point where "calling an LLM API" turns into "building a system that reasons, uses tools, and takes action" — and Azure AI Foundry Agent Service, paired with .NET, makes this surprisingly approachable. In this post, I'll walk through exactly how to stand up your first agent end-to-end — from the Azure side setup to the actual C# code — and share the full walkthrough in video form as well. 🎥 Watch the full hands-on video here: https://youtu.be/mrsEsculrNg Why Agents, and Why Now Most of us started our AI journey with a simple chat completion call — send a prompt, get text back. That's fine for Q&A, but it falls apart the moment you need the model to do something: run code, search documents, call an API, or hold a multi-turn conversation with real state. That's exactly the gap Foundry Agent Service closes. An agent in Foundry is: Durable — it lives as a resource in your Foundry project, not in your app's memory Tool-aware — it can invoke built-in tools (like a code interpreter) or your own custom functions Stateful — conversations persist and carry context across turns And the best part for us .NET folks: the entire thing is callable from clean, typed C# — no wrestling with raw REST payloads. What You'll Need Before writing any code, set up the Azure side: An Azure AI Foundry project with a chat model deployed (e.g., gpt-4o-mini ) The Foundry User RBAC role assigned to your account at the resource/resource-group scope — this is the single most common blocker people hit (a silent 403 when calling the SDK), so don't skip it az login run locally, so your code can authenticate without hardcoding any keys If you've worked with Cognitive Services roles before, note that agent management needs this separate Foundry-specific role — that trips up a lot of people coming from plain Azure OpenAI usage. Setting Up the .NET Project dotnet new console -n FoundryAgen

A few weeks back I picked up google-gemini/gemini-cli issue #28090: the shell tool was forwarding a command's entire stdout/stderr straight into the model's context, with no cap unless you opted into an LLM-based summarization step. Run one noisy build command and you'd hand the model tens of thousands of tokens of log spam it never asked for. The fix sounded trivial: cap the output before it goes into llmContent . I had a one-liner in my head before I'd even opened the file. That one-liner is exactly the kind of "obviously correct" fix that ships bugs. The one-liner The naive version looks like this: const MAX = 32 * 1024 ; // 32 KiB function truncate ( output ) { if ( output . length <= MAX ) return output ; return output . slice ( 0 , MAX ) + ' \n ...[truncated]... \n ' + output . slice ( - MAX ); } It compiles. It passes a quick manual test with a big ASCII log file. It looks done. I almost committed it as-is before writing the actual test suite. The problem is what .slice() is slicing. JavaScript strings are sequences of UTF-16 code units, not bytes and not Unicode codepoints. Most characters in typical shell output (letters, digits, punctuation) are one code unit each, so .slice() looks safe in casual testing. But the moment real-world command output contains anything outside the Basic Multilingual Plane — an emoji in a commit message, certain box-drawing/progress-bar characters some CLIs use, non-Latin filenames — that character is represented as a surrogate pair : two 16-bit code units that only mean something together. Slice between them and you don't get an error. You get one dangling unpaired surrogate on each side of the cut, silently baked into the string that gets sent to the model. No exception. No lint warning. JSON.stringify on the payload can even throw later, in a completely unrelated part of the request pipeline, for a reason that has nothing to do with where the bug actually is. Or worse: it doesn't throw, and the model just receives a slightly

Say "fable-mode" (or "operate carefully," or "high-judgment mode" or "think like fable") in a Claude Code or Codex session, and one line buried in the skill file does most of the work: "Never promote 'plausible' to 'confirmed' in your summary." That single rule is the whole philosophy in miniature — don't let something that merely looks right pass as something that was checked. Fable-mode is not a tool, a model swap, or a set of new commands. It's a behavioral overlay — a skill file at ~/.claude/skills/fable-mode/SKILL.md or ~/.codex/skills/fable-mode/SKILL.md or any other agent you use, that gets loaded into the turn and changes how the agent scopes work, verifies its own claims, and reports results. Nothing about what Agents can do changes. What changes is the discipline around using it. What actually changes The skill is organized into seven blocks, and each one targets a specific failure mode that shows up in normal agent use: Judgment before action. The first move is classification: is this a question, a problem description, or a change request? For the first two, the deliverable is an assessment — investigate and stop, don't start editing files nobody asked you to edit. Irreversible actions (deletes, pushes, overwrites) require a stop-and-confirm, and specifically: "before deleting or overwriting anything, actually look at the target." Verification as the non-negotiable core. This is the section that does the heavy lifting. Every "done" claim needs evidence produced after the last edit — a test run before your final change proves nothing about your final change. Claims get graded on a three-tier scale: Confirmed (you ran it and saw the result), Plausible (the code reads right but you haven't executed it), Assumed (inherited from the user's description). The rule is that you report the grade you actually earned, not the one that sounds better. Reasoning habits. Hold two hypotheses before committing to a diagnosis. Treat your memory of a library's API as a prior

I watched an AI agent send 200 emails to the wrong recipients because I forgot one validation check. The emails were well written. The offers were real. The recipients were just... not our leads. That was early. I learned fast. Every agent I build now has three layers of guardrails before it touches a database or an API. Here's exactly what those layers look like and why they're non-negotiable for production. Input Validation: Your Prompt Is Not a Schema The first mistake people make is trusting the LLM to produce valid output. It won't. Not reliably. I've seen GPT-4 return a JSON key called "emial" instead of "email" in a critical pipeline. One typo, and the whole record is garbage. The fix is a strict validation layer that runs before any data reaches your system. In my AI resume tailor, I use a JSON schema with conditional presence flags. Every field that must be real has a has_* boolean guard. If the LLM tries to fabricate a phone number, the schema rejects it. const resumeSchema = z . object ({ contact : z . object ({ email : z . string (). email (), phone : z . string (). optional (), has_phone : z . boolean () }). refine ( data => { // If phone is present, the guard must be true return data . phone ? data . has_phone : ! data . has_phone }, " Phone number present but has_phone flag is false " ) }) This pattern catches hallucinations before they corrupt your database. The schema is the contract. The LLM is just a suggestion engine. Permission Scoping: Give Agents the Minimum They Need An agent should never have write access to tables it doesn't need. That sounds obvious, but I've seen production systems where a job description rewriting agent had full CRUD access to the user table. When I built the LLM scoring pipeline for a job board platform, I created separate database roles. The scoring agent only had SELECT on the job listings table and INSERT on a scoring results table. It never touched users, applications, or configuration. Even if the prompt was hijack

When I first wrote about NutriAgent in November 2025, it was a full application. It had a Python backend, a web interface, a Telegram bot, user accounts, Google OAuth, Supabase, and a Google Sheets integration. Recently, I reproduced its core workflow as a skill for my personal AI agent. It took around 15 minutes and two prompts. I didn't build another backend, deploy a service, or implement OAuth again. I explained how I wanted the workflow to behave, tested it, and watched a new row appear in my nutrition spreadsheet. The original application wasn't a mistake. It was how I could deliver that experience with the tools available at the time. Eight months later, the starting point had changed. Eight Months Ago, This Was an App I built NutriAgent because I wanted to track calories and protein without trapping my data inside a nutrition app. I wanted the raw records in a spreadsheet I controlled, where I could create my own reports and eventually connect nutrition with my training data. The first version was a personal n8n workflow. It worked for me, but when a friend wanted to try it, I realized that everything was tied to my accounts. To make it reusable, I rebuilt it in Python and added the parts a real multi-user product needed: authentication, storage, a web interface, Telegram, Google OAuth, conversation history, and account linking. I've already told that story in I Ditched MyFitnessPal and Built an AI Agent to Track My Food , and later wrote about what broke after I used it every day for a month . This article starts after that version. My Gaming PC Became an Agent Box I had a modest gaming PC at home with 16 GB of RAM and a 1 TB drive. Using it through Windows, WSL, and remote desktop from my Mac felt awkward, so I installed Linux and turned it into a remote box for running agents. I'll write about that setup separately. I moved Hermes there from a VPS. Hermes is the personal agent I now run on that machine. It can load reusable skills and use tools connected

Everyone's talking about AI agents right now, but most tutorials stop at "call an LLM in a loop." If...

I used to deride the idea of an AI assistant from the moment they entered the picture (after seeing all the different *Claw variants). Why would regular people like me need an assistant? The best use case I had heard was: "Oh! It helps us decide whether I or my partner should drive the kids today!" Solving that sounded like a silly problem for an AI assistant to handle. Again, I didn't know any better because I didn't have that problem. I thought I could handle one-off tasks with just an AI subscription. What else was there? I only found the answer once I had a specific use case for it. I attended a business event, talked to a dozen people, and collected several business cards. I wanted to send each person a personalized email thanking them and continuing our conversation. If I were to do this manually, the process would look like this: open the email client, manually type in each email address from the business cards, ensure I typed everything correctly, compose my message, and again, make sure I didn't press "send" prematurely. Just thinking about it felt tedious. That is when the idea of an assistant started making sense. I fire up my coding agent(not a *Claw still), I take a single photo of all the business cards together. I ask the it to extract the names and email addresses. Then, I ask it to loop through the list and ask me what I want to send to each person. It creates drafts(which I still manually review - can't trust them enough), I say send, and then it sends them all automatically. It feels exactly like talking to a real assistant: you tell them what you want done, and it gets done without you having to press buttons or navigate a UI. That is exactly what I did; I simply gave it instructions using my voice. This makes me feel that having an AI assistant is indeed helpful. It might have also been useful to jump on this a little sooner, as I could have bought that Mac Mini at the older, lower price.

I had a rule with no exceptions. RETSBAN, my primary agent, runs local inference only: open models on my own hardware, no cloud, no fallback. If the GPU box is down, the agent is down. Then Mia, the agent that runs my marketing, needed a frontier model to do her job well. And I did something that felt strangely formal for a one-person company. I amended my own policy, in writing, with the date attached. No one was in the room Here is what took me a while to see. A human employee remembers the day you made an exception. They were in the room when you said fine, just this once. An agent was never in the room. There is no room. Every session starts cold from the files, and the files are the only memory the company has. So when the policy says local only, no exceptions, and reality contains an exception, one of two things happens. Either the agent obeys the file and blocks work I actually want done, or it notices the contradiction and starts guessing which side to trust. The guessing is the dangerous case. A rule that has been contradicted once, silently, is not a rule anymore. Every agent that loads it gets to decide, in every session, whether to believe it. You never see the deciding. You just see the drift. An exception that lives in your head does not bend a rule. It erases it. The amendment On 2026-04-23 I opened the policy file, a file literally named local_only_no_anthropic, and narrowed it instead of breaking it. The amendment names who is exempt: Mia, and only Mia. It says why: marketing work that needs capability the local stack does not have. It carries the date, and it points to the full model-topology doc for anyone, human or agent, who wants the whole picture. RETSBAN's constraint did not move an inch. Still local only. Still no fallback. Still down when the GPU box is down. That is the difference between an amendment and a repeal. An unwritten exception repeals the rule and hides the repeal. A written amendment narrows the rule and makes it stronger, beca

When you give an LLM a tool, you hand it a real function and let it choose the arguments. Those tools are everything your agent can do to a real system: read a file, write to your database, send an email, run a shell command, delete data. They are your risk surface, and most teams have never looked at it in one place. So we did. We ran scan across a batch of popular open-source TypeScript AI agents. A few of the things it found, none of them exotic: A coding agent whose grep and glob tools, which sound read-only, actually shell out through execSync . Its bash tool passes a model-chosen string straight to spawn . Arbitrary command execution, behind three innocuous names. A query tool that fires an HTTP DELETE . A "query" that deletes. A calculator that runs eval on whatever the model types, in a widely-used agent framework. Arbitrary code execution behind the friendliest name in the box. A send-email tool that posts to an array of recipients, so the model chooses who gets mailed. The single most common finding, in almost every agent we scanned: a fetch tool aimed at whatever URL the model supplies. That is a door to your internal network (an SSRF surface). Notice the pattern. The dangerous tools are not named dangerous . They are named grep , query , calculator . A name is a claim. The code is the evidence. See your own agent's tools scan reads that evidence. One command, no install, no signup, no code change: npx @agentx-core/scan . It lists every tool the model can call and ranks each one by what it can do, from read-only up to destructive: 🔍 AGENTX SCAN (TypeScript · 3 files · 5 tools) =========================================================================== RISK TOOL GUARD WHY ---- ---- ------ ------------------------ high calculator yes calls `eval` lib/tools/compute.ts:4 high grep yes calls `execSync` lib/tools/system.ts:8 med sendEmail yes calls `mailer.send` lib/tools/io.ts:5 med fetchUrl yes outbound req to agent-controlled host (SSRF) lib/tools/io.ts:11 2