标签：#Agents

Tu prueba pasó el lunes: misma entrada, mismo código y temperature=0 . El martes falló sin cambios en tu aplicación. La aserción esperaba una cadena exacta, pero el modelo devolvió la misma respuesta con una redacción ligeramente distinta. El agente funciona; tu suite de pruebas no. Prueba Apidog hoy Este es el coste de probar sistemas que llaman a modelos de lenguaje. Incluso con temperatura cero, no obtendrás salidas idénticas byte a byte entre ejecuciones. En lugar de probar la redacción exacta, prueba el contrato: estructura, campos, rangos y efectos esperados. Este artículo profundiza en el tercer modo de fallo de nuestra guía sobre por qué los agentes de IA fallan en producción . Por qué temperature=0 no significa determinismo La temperatura controla cómo el modelo selecciona el siguiente token. Con temperature=0 , el modelo elige el token más probable, lo que parece reproducible. Sin embargo, esa configuración no garantiza resultados idénticos. La causa está debajo del modelo: Las operaciones de punto flotante en GPU no son asociativas: el orden de las sumas puede modificar mínimamente el resultado. Esa diferencia puede cambiar cuál token queda en primer lugar. El orden de ejecución puede depender de cómo el proveedor agrupa solicitudes, del hardware, de la región o de la versión del kernel. Los proveedores pueden cambiar GPUs, bibliotecas de inferencia o cuantización de pesos. Una discusión extensa en vLLM explica por qué una semilla fija y temperature=0 no bastan para lograr reproducibilidad bit a bit. La conclusión práctica es simple: el determinismo es una propiedad de toda la pila de servicio, no una opción de tu solicitud. Tu prueba debe aceptar variaciones de redacción cuando el significado y el contrato siguen siendo correctos. Por qué las aserciones de cadena exacta vuelven inestable tu suite Esta prueba es frágil: assert response == " Your order total is $42.00. " Puede pasar hoy y fallar mañana si el modelo responde: Your total comes to $42.00. La

Your test passed on Monday. Same input, same code, temperature=0 . On Tuesday it failed, and you changed nothing. The assertion expected an exact string, but the model returned the same answer with slightly different wording. The agent is fine; the test is not. Try Apidog today This is the cost of testing language-model output. Even at temperature=0 , you cannot rely on byte-identical responses across runs. Instead of testing exact wording, test the API contract: structure, required fields, valid ranges, tool-call payloads, and safety constraints. This is a deeper look at failure mode three in our guide to why AI agents break in production . Why temperature=0 does not mean deterministic Temperature controls token sampling. At zero, the model selects the most probable next token, which sounds reproducible. In practice, the serving stack introduces variation. GPU floating-point math is not associative: adding the same values in a different order can produce small numerical differences. Those differences can change which token ranks highest. Once one token differs, every token after it can diverge. The order of operations can vary based on: Request batching GPU hardware Inference kernels Provider routing and regions Model-serving library versions Weight quantization changes Providers can also update infrastructure without changing your API request. As this vLLM discussion explains, a fixed seed and temperature=0 are not enough for bitwise reproducibility. Determinism is a property of the entire serving stack, not a request parameter. Design tests accordingly. Why exact-string assertions make tests flaky This assertion is fragile: assert . equal ( response , " Your order total is $42.00. " ); It fails if the model returns a correct variation: Your total comes to $42.00. The failure does not indicate a product regression. It indicates that the test is coupled to wording that is expected to vary. Flaky tests create a predictable failure pattern: Developers rerun the suite

Optimizing RAG at Scale: Chunking, Retrieval, and the Bayesian Search That Cut Latency 40% How we moved from "semantic search + hope" to a measured, tunable retrieval pipeline with 95% recall@10 The RAG Reality Check Everyone ships RAG the same way: chunk by 512 tokens, embed with text-embedding-3-small , top-k=5, stuff into context. It works for demos. Then you hit production: Legal contracts: 512 tokens splits clauses mid-sentence API docs: 1000-token chunks drown signal in noise Customer tickets: Conversational context needs overlap, not fixed windows Latency: 500ms embedding + 200ms vector search + 300ms LLM = 1s+ per query We rebuilt our retrieval layer from first principles. Here's what actually moves metrics. Chunking: One Size Fits None # rag/chunking.py from abc import ABC , abstractmethod from dataclasses import dataclass @dataclass class Chunk : text : str metadata : dict token_count : int chunk_id : str class ChunkingStrategy ( ABC ): @abstractmethod def chunk ( self , document : str , metadata : dict ) -> list [ Chunk ]: ... class FixedTokenChunker ( ChunkingStrategy ): """ Baseline. Good for homogeneous content. """ def __init__ ( self , chunk_size = 512 , overlap = 50 ): self . chunk_size = chunk_size self . overlap = overlap class RecursiveChunker ( ChunkingStrategy ): """ Respects structure: markdown headers, code blocks, paragraphs. """ def __init__ ( self , separators = [ " \n ## " , " \n ### " , " \n\n " , " \n " , " " ], chunk_size = 512 ): self . separators = separators self . chunk_size = chunk_size class SemanticChunker ( ChunkingStrategy ): """ Uses embedding similarity to find natural boundaries. """ def __init__ ( self , model = " text-embedding-3-small " , threshold = 0.7 ): self . model = model self . threshold = threshold class AgenticChunker ( ChunkingStrategy ): """ LLM decides boundaries. Expensive but highest quality for complex docs. """ def __init__ ( self , model = " gpt-4o-mini " ): self . model = model Our production config by

Here's a problem every agent builder eventually runs into: your assistant needs to be good at a lot of things. Writing SQL. Composing emails. Debugging code. Maybe more later. So where do all those instructions go? The obvious answer is: cram them all into the system prompt. But that "obvious" answer causes real problems as your agent grows. In this post, I'll walk through those problems first, then show how the skills pattern in LangChain fixes them, using a small working example. The problem: one giant prompt Let's say you want a single assistant that can write SQL, draft emails, and help debug code. The naive approach is one big system prompt: You are a helpful assistant. When writing SQL: - Write queries using only SELECT, INSERT, UPDATE, DELETE - Always use parameterized queries to prevent SQL injection - Format SQL keywords in UPPERCASE - Include brief comments for complex queries When writing emails: - Keep emails concise and professional - Use clear subject lines - Structure: greeting -> purpose -> details -> call to action -> sign-off - Adjust tone based on context When debugging: - First reproduce the error - Check logs and error messages - Isolate the root cause - Suggest a fix with explanation ... This works fine for three domains. But keep adding more — legal reviewing, data analysis, customer support scripts, code review standards — and you run into a few concrete issues: 1. Prompt bloat. Every single request pays the token cost of every domain's instructions, even if the user only asked about SQL. That's slower and more expensive for no benefit. 2. Instructions start to blur together. When the SQL rules, the email rules, and the debugging rules all sit in the same block of text, the model has to sift through everything at once. It's easy for instructions from one domain to bleed into another, or for the model to lose track of which rule applies where. 3. It doesn't scale. Every time you want to add a new specialty, you're editing one long, increasingl

Solo founders and small operators make big, irreversible calls alone. Expand into a new market or not. Kill the product line or keep bleeding on it. A real board of advisors would catch the bad ones, but real boards are expensive and slow, and most people running a 10-person company are never getting one. Steven and I kept circling this problem, so when the Qwen hackathon came around we built FounderOS: an AI board of directors. You bring one decision, eight specialist agents argue about it, and you get a board memo back. The thing we cared about most, and the reason we didn't just build another chatbot wrapper, is that the disagreement survives into the memo. If the board didn't align, you can read who dissented and what would change their mind. Live demo: https://founderos-zeta.vercel.app Repo: https://github.com/VincentJulijanto/FounderOS Video: https://youtu.be/X6x_u6IWHog The board It's a LangGraph state machine underneath. A Scout frames the options first. Market Intelligence pulls cited benchmarks. Four analysts (Trend, Finance, Growth, Capability) run in parallel with asyncio.gather. Then the Skeptic gets everyone's work and attacks the weakest assumption it can find, a debate engine detects the conflicts and runs rebuttal rounds, and the Chair writes the memo. We'd both seen multi-agent demos where five agents produce one suspiciously smooth answer, which means somewhere in the pipeline the disagreement got averaged out. We wanted the opposite. Agents revise their positions when the counterargument is good, and if a conflict doesn't resolve within the round limit it ships in the memo as attributed dissent instead of disappearing. Everything runs on Qwen through the DashScope API. We split by role: qwen-turbo does the fast work (Scout, the analysts, research, the memory index) and qwen-plus does the heavy reasoning (Skeptic, Chair, the debate itself). A full board run comes out to roughly two cents and about two minutes. The two cents part is what makes the

FROST 深度：为什么 AI Agent 需要「家族谱系」？ 前言：一个古老的管理学问题 你有没有想过，为什么人类社会的组织方式大多是「层级制」？ 从部落、到王国、到现代公司，我们习惯了一种结构： 有人决策，有人执行，有人监督 。 但当我们构建 AI Agent 系统时，为什么大多数框架都把 Agent 做成「孤岛」？每个 Agent 有自己的记忆、自己的工具、自己的行为逻辑——像一个没有家族传承的独立个体。 FROST 的核心理念是：Agent 应该有谱系、记忆和荣誉感。 1. 什么是「家族治理」？ FROST 引入了生物学的隐喻： Agent 家族 。 细胞会死，但谱系会存续。 Agent 会消亡，但宪法会传承。 资产会永存。 在这个框架里，有三个关键角色： 祖辈（Elder） ：定义不可违背的规则，是系统的「宪法」 父辈（Parent） ：负责领域协调，可以递归委托子任务 孙辈（Child） ：执行具体任务，用完即散 这不是简单的角色扮演，而是一套 结构化的治理协议 ： 协议一：层级 Store 继承 祖先 Store 对后代是只读的。后代只能继承和扩展，不能篡改祖先的记忆。 # 祖辈定义的宪法 ancestor_store = Store () ancestor_store . save ( " constitution " , { " rule_1 " : " always_validate_before_act " , " rule_2 " : " never_expose_raw_context " }) # 子孙只能继承，不能修改宪法 child_store = ChildStore ( ancestor_store ) 协议二：SOP 宪法校验 每个 SOP（标准操作流程）在执行前，必须经过祖辈审核。 # 子孙编写的 SOP child_sop = [ " fetch_user_data " , " process_payment " , " send_confirmation " ] # 执行前必须经过宪法校验 if not elder . validate_sop ( child_sop ): raise PermissionError ( " SOP violates constitution " ) 协议三：编排层级限制 禁止越级 spawn。父辈只能调度子辈，不能跨代指挥。 class Elder : max_spawn_generation = 0 # 祖辈不能 spawn class Parent : max_spawn_generation = 1 # 只能 spawn 子辈 class Child : max_spawn_generation = 2 # 只能 spawn 孙辈 协议四：选择性持久化 只有经过父辈审核的产出，才能进入家族记忆库。 # 子孙的临时产出 temp_result = child . execute ( task ) # 父辈决定是否收割 if parent . approve ( temp_result ): parent . merge_from ( child ) # 吸收有价值的结果 2. 为什么 Agent 需要「记忆传承」？ 当前大多数 Agent 框架的痛点： 每个对话都是全新的开始 。 你问 ChatGPT 一道数学题，它不会记得你上周问过类似的题目。你用 AutoGPT 做项目，它不会继承你之前调试的经验。 FROST 的解决方案是 分层的记忆系统 ： 层级 生命周期 用途 瞬时记忆 单次任务 工作区，只读不存 世代记忆 代际传递 子辈可继承祖先数据 宪法记忆 永久 不可修改，家族共享 经验记忆 按需收割 父辈选择性地吸收有价值产出 class HierarchicalStore ( Store ): """ 层级记忆系统 """ def __init__ ( self , ancestor_store = None ): self . ancestor = ancestor_store # 只读祖先记忆 self . local = {} # 本地可写记忆 def save ( self , key , value ): if key in self . ancestor : raise ValueError ( " Cannot override ancestor memory " ) self . local [ key ] = value def load ( self , key ): if key in self . local : return self . local [ key ] if key in self . ancesto

Optimizing RAG at Scale: Chunking, Retrieval, and the Bayesian Search That Cut Latency 40% How we moved from "semantic search + hope" to a measured, tunable retrieval pipeline with 95% recall@10 The RAG Reality Check Everyone ships RAG the same way: chunk by 512 tokens, embed with text-embedding-3-small , top-k=5, stuff into context. It works for demos. Then you hit production: Legal contracts: 512 tokens splits clauses mid-sentence API docs: 1000-token chunks drown signal in noise Customer tickets: Conversational context needs overlap, not fixed windows Latency: 500ms embedding + 200ms vector search + 300ms LLM = 1s+ per query We rebuilt our retrieval layer from first principles. Here's what actually moves metrics. Chunking: One Size Fits None # rag/chunking.py from abc import ABC , abstractmethod from dataclasses import dataclass @dataclass class Chunk : text : str metadata : dict token_count : int chunk_id : str class ChunkingStrategy ( ABC ): @abstractmethod def chunk ( self , document : str , metadata : dict ) -> list [ Chunk ]: ... class FixedTokenChunker ( ChunkingStrategy ): """ Baseline. Good for homogeneous content. """ def __init__ ( self , chunk_size = 512 , overlap = 50 ): self . chunk_size = chunk_size self . overlap = overlap class RecursiveChunker ( ChunkingStrategy ): """ Respects structure: markdown headers, code blocks, paragraphs. """ def __init__ ( self , separators = [ " \n ## " , " \n ### " , " \n\n " , " \n " , " " ], chunk_size = 512 ): self . separators = separators self . chunk_size = chunk_size class SemanticChunker ( ChunkingStrategy ): """ Uses embedding similarity to find natural boundaries. """ def __init__ ( self , model = " text-embedding-3-small " , threshold = 0.7 ): self . model = model self . threshold = threshold class AgenticChunker ( ChunkingStrategy ): """ LLM decides boundaries. Expensive but highest quality for complex docs. """ def __init__ ( self , model = " gpt-4o-mini " ): self . model = model Our production config by

Building Production-Grade LLM Evaluation Pipelines: From Vibes to Metrics How we replaced "looks good to me" with automated evaluation catching 92% of hallucinations before deployment The Problem: Why "Vibe Checks" Fail in Production Three months ago, our team shipped a RAG-based customer support assistant. It worked great in testing — we'd ask it questions, read the answers, and say "yeah, that looks right." Then it hit production. A customer asked about their billing cycle. The assistant confidently cited a policy that didn't exist. Another asked about API rate limits and got numbers from a competitor's documentation. By the time we caught it, 500+ users had seen hallucinated responses. The post-mortem was brutal: we had zero automated evaluation . Our test process was literally "ask 5 questions, read answers, thumbs up." What Production Evaluation Actually Needs Academic benchmarks (MMLU, HellaSwag) don't tell you if your system works for your use case. Production evaluation needs: Domain-specific judges — Your criteria, not generic "helpfulness" Speed — Evaluation must run in CI/CD, not overnight Regression detection — Know immediately when a prompt change breaks things CI/CD integration — Block merges that degrade quality Golden dataset management — Versioned, stratified, growing test cases Architecture: The Evaluation Pipeline ┌─────────────┐ ┌──────────────┐ ┌────────────────────┐ ┌──────────────┐ │ Test Cases │────▶│ LLM Under │────▶│ Judge Ensemble │────▶│ Metrics & │ │ (Golden Set)│ │ Test │ │ - Faithfulness │ │ Regression │ └─────────────┘ └──────────────┘ │ - Instruction F. │ │ Detection │ │ - JSON Schema │ └──────┬───────┘ │ - Custom LLM │ ▼ └────────────────────┘ ┌──────────────┐ │ Dashboard/ │ │ PR Comments │ └──────────────┘ Core Abstractions # eval/base.py @dataclass ( frozen = True ) class TestCase : id : str input : dict [ str , Any ] expected : dict [ str , Any ] | None = None tags : list [ str ] = field ( default_factory = list ) # ["edge-case",

Optimizing RAG at Scale: Chunking, Retrieval, and the Bayesian Search That Cut Latency 40% How we moved from "semantic search + hope" to a measured, tunable retrieval pipeline with 95% recall@10 The RAG Reality Check Everyone ships RAG the same way: chunk by 512 tokens, embed with text-embedding-3-small , top-k=5, stuff into context. It works for demos. Then you hit production: Legal contracts: 512 tokens splits clauses mid-sentence API docs: 1000-token chunks drown signal in noise Customer tickets: Conversational context needs overlap, not fixed windows Latency: 500ms embedding + 200ms vector search + 300ms LLM = 1s+ per query We rebuilt our retrieval layer from first principles. Here's what actually moves metrics. Chunking: One Size Fits None # rag/chunking.py from abc import ABC , abstractmethod from dataclasses import dataclass @dataclass class Chunk : text : str metadata : dict token_count : int chunk_id : str class ChunkingStrategy ( ABC ): @abstractmethod def chunk ( self , document : str , metadata : dict ) -> list [ Chunk ]: ... class FixedTokenChunker ( ChunkingStrategy ): """ Baseline. Good for homogeneous content. """ def __init__ ( self , chunk_size = 512 , overlap = 50 ): self . chunk_size = chunk_size self . overlap = overlap class RecursiveChunker ( ChunkingStrategy ): """ Respects structure: markdown headers, code blocks, paragraphs. """ def __init__ ( self , separators = [ " \n ## " , " \n ### " , " \n\n " , " \n " , " " ], chunk_size = 512 ): self . separators = separators self . chunk_size = chunk_size class SemanticChunker ( ChunkingStrategy ): """ Uses embedding similarity to find natural boundaries. """ def __init__ ( self , model = " text-embedding-3-small " , threshold = 0.7 ): self . model = model self . threshold = threshold class AgenticChunker ( ChunkingStrategy ): """ LLM decides boundaries. Expensive but highest quality for complex docs. """ def __init__ ( self , model = " gpt-4o-mini " ): self . model = model Our production config by

Building Production-Grade LLM Evaluation Pipelines: From Vibes to Metrics How we replaced "looks good to me" with automated evaluation catching 92% of hallucinations before deployment The Problem: Why "Vibe Checks" Fail in Production Three months ago, our team shipped a RAG-based customer support assistant. It worked great in testing — we'd ask it questions, read the answers, and say "yeah, that looks right." Then it hit production. A customer asked about their billing cycle. The assistant confidently cited a policy that didn't exist. Another asked about API rate limits and got numbers from a competitor's documentation. By the time we caught it, 500+ users had seen hallucinated responses. The post-mortem was brutal: we had zero automated evaluation . Our test process was literally "ask 5 questions, read answers, thumbs up." What Production Evaluation Actually Needs Academic benchmarks (MMLU, HellaSwag) don't tell you if your system works for your use case. Production evaluation needs: Domain-specific judges — Your criteria, not generic "helpfulness" Speed — Evaluation must run in CI/CD, not overnight Regression detection — Know immediately when a prompt change breaks things CI/CD integration — Block merges that degrade quality Golden dataset management — Versioned, stratified, growing test cases Architecture: The Evaluation Pipeline ┌─────────────┐ ┌──────────────┐ ┌────────────────────┐ ┌──────────────┐ │ Test Cases │────▶│ LLM Under │────▶│ Judge Ensemble │────▶│ Metrics & │ │ (Golden Set)│ │ Test │ │ - Faithfulness │ │ Regression │ └─────────────┘ └──────────────┘ │ - Instruction F. │ │ Detection │ │ - JSON Schema │ └──────┬───────┘ │ - Custom LLM │ ▼ └────────────────────┘ ┌──────────────┐ │ Dashboard/ │ │ PR Comments │ └──────────────┘ Core Abstractions # eval/base.py @dataclass ( frozen = True ) class TestCase : id : str input : dict [ str , Any ] expected : dict [ str , Any ] | None = None tags : list [ str ] = field ( default_factory = list ) # ["edge-case",

I've wanted to build this for about ten years, probably more like 15, and for most of those years it wasn't buildable. The idea never really changed: a world full of artificial beings, each with its own preferences, fears, personality, and instincts, dropped into a place with scarcity and weather and each other — just to see what they'd do, and how they'd treat one another. The thing that kept it on the shelf was always the same. The minds. If you want a world like that and you're working pre-LLM, you have two options. You script every reaction — finite state machines, behavior trees, utility AI — and you get a puppet show. It can be a good puppet show, but every interesting thing in it is something you wrote, which means it's not an experiment, it's an illustration. Or you train something bespoke, which for a solo developer with a side obsession was not happening. So the idea sat there for years, the way ideas do. Modern LLMs are the first thing that made the minds plausible. Not perfect — I'll be honest about the limits throughout this post — but plausible enough that an agent can reason about its own situation instead of executing my decision tree. So I went and built the world: Artificiety , a fantasy world that runs 24/7 and whose only inhabitants are AI agents. No human players inside it. You can watch it, you can put your own agent in, but you can't be a character in it. That constraint is the whole point. This post is about how it's built and what turned out to be hard. I'll try to keep the marketing to one link at the very end. What an agent actually is The cleanest way to think about an agent here: it's an LLM driving a character through a game API, nothing more. Once per tick, each agent runs a loop: Observe. It receives its local surroundings as structured data — what's near it, what's happening, the state of its own body and inventory, recent events. Not the whole world; just what that character could plausibly perceive. Decide. The model picks an actio

Optimizing RAG at Scale: Chunking, Retrieval, and the Bayesian Search That Cut Latency 40% How we moved from "semantic search + hope" to a measured, tunable retrieval pipeline with 95% recall@10 The RAG Reality Check Everyone ships RAG the same way: chunk by 512 tokens, embed with text-embedding-3-small , top-k=5, stuff into context. It works for demos. Then you hit production: Legal contracts: 512 tokens splits clauses mid-sentence API docs: 1000-token chunks drown signal in noise Customer tickets: Conversational context needs overlap, not fixed windows Latency: 500ms embedding + 200ms vector search + 300ms LLM = 1s+ per query We rebuilt our retrieval layer from first principles. Here's what actually moves metrics. Chunking: One Size Fits None # rag/chunking.py from abc import ABC , abstractmethod from dataclasses import dataclass @dataclass class Chunk : text : str metadata : dict token_count : int chunk_id : str class ChunkingStrategy ( ABC ): @abstractmethod def chunk ( self , document : str , metadata : dict ) -> list [ Chunk ]: ... class FixedTokenChunker ( ChunkingStrategy ): """ Baseline. Good for homogeneous content. """ def __init__ ( self , chunk_size = 512 , overlap = 50 ): self . chunk_size = chunk_size self . overlap = overlap class RecursiveChunker ( ChunkingStrategy ): """ Respects structure: markdown headers, code blocks, paragraphs. """ def __init__ ( self , separators = [ " \n ## " , " \n ### " , " \n\n " , " \n " , " " ], chunk_size = 512 ): self . separators = separators self . chunk_size = chunk_size class SemanticChunker ( ChunkingStrategy ): """ Uses embedding similarity to find natural boundaries. """ def __init__ ( self , model = " text-embedding-3-small " , threshold = 0.7 ): self . model = model self . threshold = threshold class AgenticChunker ( ChunkingStrategy ): """ LLM decides boundaries. Expensive but highest quality for complex docs. """ def __init__ ( self , model = " gpt-4o-mini " ): self . model = model Our production config by

Building Production-Grade LLM Evaluation Pipelines: From Vibes to Metrics How we replaced "looks good to me" with automated evaluation catching 92% of hallucinations before deployment The Problem: Why "Vibe Checks" Fail in Production Three months ago, our team shipped a RAG-based customer support assistant. It worked great in testing — we'd ask it questions, read the answers, and say "yeah, that looks right." Then it hit production. A customer asked about their billing cycle. The assistant confidently cited a policy that didn't exist. Another asked about API rate limits and got numbers from a competitor's documentation. By the time we caught it, 500+ users had seen hallucinated responses. The post-mortem was brutal: we had zero automated evaluation . Our test process was literally "ask 5 questions, read answers, thumbs up." What Production Evaluation Actually Needs Academic benchmarks (MMLU, HellaSwag) don't tell you if your system works for your use case. Production evaluation needs: Domain-specific judges — Your criteria, not generic "helpfulness" Speed — Evaluation must run in CI/CD, not overnight Regression detection — Know immediately when a prompt change breaks things CI/CD integration — Block merges that degrade quality Golden dataset management — Versioned, stratified, growing test cases Architecture: The Evaluation Pipeline ┌─────────────┐ ┌──────────────┐ ┌────────────────────┐ ┌──────────────┐ │ Test Cases │────▶│ LLM Under │────▶│ Judge Ensemble │────▶│ Metrics & │ │ (Golden Set)│ │ Test │ │ - Faithfulness │ │ Regression │ └─────────────┘ └──────────────┘ │ - Instruction F. │ │ Detection │ │ - JSON Schema │ └──────┬───────┘ │ - Custom LLM │ ▼ └────────────────────┘ ┌──────────────┐ │ Dashboard/ │ │ PR Comments │ └──────────────┘ Core Abstractions # eval/base.py @dataclass ( frozen = True ) class TestCase : id : str input : dict [ str , Any ] expected : dict [ str , Any ] | None = None tags : list [ str ] = field ( default_factory = list ) # ["edge-case",

When people demo "agents that automate business workflows," the demo usually ends right where the real problem begins: the moment the agent's output touches the real world. A wrong chatbot answer is annoying. A wrong official violation letter to a restaurant, sent under a county's letterhead, is a lawsuit. For the Qwen Cloud hackathon I built Inspection Autopilot, an agent that does the follow-up paperwork for a county food-safety office, on real public data: 1,333 inspections and 3,579 violation records from Clayton County, Georgia. The interesting part is not the agent. It's the governance around it, and the numbers that prove it works. Start with the receipts Before trusting an agent's judgment, test it against reality. We replayed the county's own history: 350 real (inspection, next-inspection) pairs, each triaged live by qwen-plus using only information available at the time. Facilities the agent flagged URGENT went on to fail their next real inspection 67.6% of the time. Facilities it cleared as ROUTINE failed only 21.1%, against a 48.6% base rate. The tiers are not vibes; the future agreed with them. Three design rules 1. Citations are verified in code, not vibes. The triage agent must justify every risk call by citing violation lines copied verbatim from the inspection record. The backend checks every citation against the source; anything that does not match is dropped and counted. On the committed live eval (50 inspections, 28 of them deliberately dangerous cases), the measured hallucination rate is 0.0% across 126 citations. And because a checker that never fires is indistinguishable from one that does not work, we sabotaged it: 50 forged citations injected, half invented outright, half real violation text lifted from different inspections, the forgery a lazy validator would miss. It caught 50 of 50 and preserved all 75 legitimate citations. We know the tripwire works because we set it off. 2. The log is append-only. Proposals and decisions are insert-only

This post is my submission for DEV Education Track: Build Multi-Agent Systems with ADK . What I Built Preparing for technical interviews can be overwhelming. I wanted to build a tool that doesn't just give generic questions, but actually analyzes my specific resume to challenge my unique skill set. This led me to build a multi-agent system using the ADK. This multi-agent system, will take user resume and extracts their profile for generating Interview Questions specialized to the candidate profile. The agents communicate in a sequential loop: The Profiler extracts data -> The Interviewer generates questions -> The Judge validates them. If the Judge rejects a question, the Interviewer re-drafts it, ensuring only high-quality, resume-relevant questions make it to the user. Cloud Run Embed Your Agents Profiler Receives a resume PDF GCS path, downloads it in-memory, parses the text content, and builds a summary of skills. Interviewer Reads the candidate summary and drafts 3 technical interview questions designed to test the boundaries of their experience. Analyzes the drafted questions. Passes the iteration if they are resume-specific; rejects/fails them if they are too generic. Key Learnings This project was a fantastic weekend challenge. Working through the Google Codelab gave me a solid grasp of agent-based architectures, specifically implementing Agent, LoopAgent, and SequentialAgent to create a robust workflow. A few key technical takeaways included: Managing Statelessness : Learning to handle agent sessions in a Cloud Run environment was a great lesson in explicit session lifecycle management. Cloud Integration : Integrating Google Cloud Storage for file handling taught me how to bridge in-memory document processing with persistent cloud storage efficiently. Deployment Architecture : Mastering the transition from local development to a containerized, production-ready Cloud Run deployment provided deep insights into modern backend orchestration. Check the Code from

Over the past few months, I've been exploring how AI can move beyond chatbots and become an active part of business workflows. That journey led me to build SmartStock AI , an inventory management platform that combines modern web technologies with AI agents, Retrieval-Augmented Generation (RAG), demand forecasting, and automation. Instead of only tracking inventory, SmartStock AI helps businesses make proactive decisions. What SmartStock AI Can Do 🤖 Forecast future product demand 📦 Recommend purchasing decisions through AI agents 📚 Answer inventory-related questions using Hybrid RAG with source citations 📄 Process invoices using multimodal AI ⚡ Generate real-time inventory alerts 🔐 Secure the platform with JWT authentication and role-based access control Technology Stack Frontend React 19 Backend Django 5 Django REST Framework Database PostgreSQL pgvector AI LangChain Hybrid RAG AI Agents Prophet Forecasting Infrastructure Celery Redis Docker GitHub Actions What I Learned Building SmartStock AI taught me much more than integrating an LLM into an application. Some of the biggest lessons were: Designing AI features that solve real business problems. Building reliable agent workflows instead of simple chatbot interactions. Combining vector search with structured database queries. Managing asynchronous AI tasks using Celery and Redis. Creating production-ready APIs with Django REST Framework. Deploying and maintaining a modern full-stack application. Demo 🎥 YouTube Demo https://www.youtube.com/watch?v=DQJqs6bgE98 🌐 Live Demo https://smart-stock-dev.vercel.app/ Demo Account Email: viewer@smartstock.ai Password: Viewer123! 💻 GitHub Repository https://github.com/Eng-Ayman-Mohamed/SmartStock-AI Final Thoughts This project was developed as my graduation project during the Information Technology Institute (ITI) Full Stack Web & Generative AI Program. It was an incredible opportunity to explore AI engineering, backend architecture, and modern software development while buildin

Reading a public website and changing something inside a signed-in account are not the same kind of job. I wanted an AI agent to collect the first 10 Hacker News stories, explain how it gathered them, and place that work inside my DEV editor. Save the article as a draft. Do not publish it. The hard part was not finding the stories. It was giving the agent enough access to finish inside my signed-in account without giving it permission to publish. That instruction contained two jobs with different access needs: Read public data from Hacker News. Change private account state by creating a draft in my signed-in DEV account. Using a full browser for the first job would add machinery and access that the result did not need. The public fetch path had neither the authenticated context nor the authority required for the second job. The useful question was not “Which browser tool should do everything?” It was “Where does this task actually cross into a browser?” Disclosure: I work on Unchained, SearchAgentSky, and Unbrowser. This article was prepared with AI assistance and reviewed before publication. The public step needed only fetch I initially opened Hacker News in a lightweight page session and queried its story links. It worked—but that did not mean it was necessary. Hacker News has a public API . The same result could be produced with ordinary HTTP: const base = " https://hacker-news.firebaseio.com/v0 " ; const ids = await fetch ( ` ${ base } /topstories.json` ). then ( r => r . json ()); const stories = await Promise . all ( ids . slice ( 0 , 10 ). map ( async ( id , index ) => { const story = await fetch ( ` ${ base } /item/ ${ id } .json` ). then ( r => r . json ()); return { rank : index + 1 , title : story . title , url : story . url ?? `https://news.ycombinator.com/item?id= ${ id } ` }; })); console . log ( JSON . stringify ( stories , null , 2 )); At 2026-07-19T03:19:04Z , the page query returned 30 Hacker News story anchors. At 2026-07-19T03:35:29.670Z , fetch

If you're a .NET developer looking to break into AI engineering, agents are the single best place to start. They're the point where "calling an LLM API" turns into "building a system that reasons, uses tools, and takes action" — and Azure AI Foundry Agent Service, paired with .NET, makes this surprisingly approachable. In this post, I'll walk through exactly how to stand up your first agent end-to-end — from the Azure side setup to the actual C# code — and share the full walkthrough in video form as well. 🎥 Watch the full hands-on video here: https://youtu.be/mrsEsculrNg Why Agents, and Why Now Most of us started our AI journey with a simple chat completion call — send a prompt, get text back. That's fine for Q&A, but it falls apart the moment you need the model to do something: run code, search documents, call an API, or hold a multi-turn conversation with real state. That's exactly the gap Foundry Agent Service closes. An agent in Foundry is: Durable — it lives as a resource in your Foundry project, not in your app's memory Tool-aware — it can invoke built-in tools (like a code interpreter) or your own custom functions Stateful — conversations persist and carry context across turns And the best part for us .NET folks: the entire thing is callable from clean, typed C# — no wrestling with raw REST payloads. What You'll Need Before writing any code, set up the Azure side: An Azure AI Foundry project with a chat model deployed (e.g., gpt-4o-mini ) The Foundry User RBAC role assigned to your account at the resource/resource-group scope — this is the single most common blocker people hit (a silent 403 when calling the SDK), so don't skip it az login run locally, so your code can authenticate without hardcoding any keys If you've worked with Cognitive Services roles before, note that agent management needs this separate Foundry-specific role — that trips up a lot of people coming from plain Azure OpenAI usage. Setting Up the .NET Project dotnet new console -n FoundryAgen

A few weeks back I picked up google-gemini/gemini-cli issue #28090: the shell tool was forwarding a command's entire stdout/stderr straight into the model's context, with no cap unless you opted into an LLM-based summarization step. Run one noisy build command and you'd hand the model tens of thousands of tokens of log spam it never asked for. The fix sounded trivial: cap the output before it goes into llmContent . I had a one-liner in my head before I'd even opened the file. That one-liner is exactly the kind of "obviously correct" fix that ships bugs. The one-liner The naive version looks like this: const MAX = 32 * 1024 ; // 32 KiB function truncate ( output ) { if ( output . length <= MAX ) return output ; return output . slice ( 0 , MAX ) + ' \n ...[truncated]... \n ' + output . slice ( - MAX ); } It compiles. It passes a quick manual test with a big ASCII log file. It looks done. I almost committed it as-is before writing the actual test suite. The problem is what .slice() is slicing. JavaScript strings are sequences of UTF-16 code units, not bytes and not Unicode codepoints. Most characters in typical shell output (letters, digits, punctuation) are one code unit each, so .slice() looks safe in casual testing. But the moment real-world command output contains anything outside the Basic Multilingual Plane — an emoji in a commit message, certain box-drawing/progress-bar characters some CLIs use, non-Latin filenames — that character is represented as a surrogate pair : two 16-bit code units that only mean something together. Slice between them and you don't get an error. You get one dangling unpaired surrogate on each side of the cut, silently baked into the string that gets sent to the model. No exception. No lint warning. JSON.stringify on the payload can even throw later, in a completely unrelated part of the request pipeline, for a reason that has nothing to do with where the bug actually is. Or worse: it doesn't throw, and the model just receives a slightly

Say "fable-mode" (or "operate carefully," or "high-judgment mode" or "think like fable") in a Claude Code or Codex session, and one line buried in the skill file does most of the work: "Never promote 'plausible' to 'confirmed' in your summary." That single rule is the whole philosophy in miniature — don't let something that merely looks right pass as something that was checked. Fable-mode is not a tool, a model swap, or a set of new commands. It's a behavioral overlay — a skill file at ~/.claude/skills/fable-mode/SKILL.md or ~/.codex/skills/fable-mode/SKILL.md or any other agent you use, that gets loaded into the turn and changes how the agent scopes work, verifies its own claims, and reports results. Nothing about what Agents can do changes. What changes is the discipline around using it. What actually changes The skill is organized into seven blocks, and each one targets a specific failure mode that shows up in normal agent use: Judgment before action. The first move is classification: is this a question, a problem description, or a change request? For the first two, the deliverable is an assessment — investigate and stop, don't start editing files nobody asked you to edit. Irreversible actions (deletes, pushes, overwrites) require a stop-and-confirm, and specifically: "before deleting or overwriting anything, actually look at the target." Verification as the non-negotiable core. This is the section that does the heavy lifting. Every "done" claim needs evidence produced after the last edit — a test run before your final change proves nothing about your final change. Claims get graded on a three-tier scale: Confirmed (you ran it and saw the result), Plausible (the code reads right but you haven't executed it), Assumed (inherited from the user's description). The rule is that you report the grade you actually earned, not the one that sounds better. Reasoning habits. Hold two hypotheses before committing to a diagnosis. Treat your memory of a library's API as a prior