标签：#agents

most people know Mistral for its chat models. the part i find more interesting for enterprise work is the Agents API : persistent agents with instructions and tools, stateful conversations you can resume, built-in connectors (web search, code interpreter, document library), and handoffs so one agent can delegate to another. the .net story stops short of this. the community sdks (tghamm's is genuinely good) cover chat completions, embeddings and function calling. they don't cover the agentic layer. so if you're a .net shop that wants to build a multi-agent workflow on mistral, you're writing raw http. i didn't want to, so i built Mistral.Agents.Net . here's the design and the one wire-format detail that cost me a debugging session. agents, not just completions a chat completion is stateless: you send messages, you get a reply, you manage all the history yourself. an agent is a stored object with instructions and tools, and a conversation is a stateful thread you can continue by id. that difference matters for enterprise workflows, where a "session" spans many turns and you want the platform to hold the state. var agent = await client . CreateAgentAsync ( new CreateAgentRequest { Model = "mistral-medium-latest" , Name = "Financial Analyst" , Instructions = "Use the code interpreter for math and web search for current facts." , Tools = { AgentTool . CodeInterpreter (), AgentTool . WebSearch () }, }); using var turn = await client . StartConversationAsync ( new StartConversationRequest { AgentId = agent . Id , Inputs = "what was 15% of last quarter's revenue if it was 12.4M?" , }); Console . WriteLine ( turn . OutputText ); the response isn't a single message. it's a list of outputs: tool executions, message chunks, function calls, handoffs. the library gives you OutputText for the common case and Outputs for the raw stream, plus a Root JsonElement escape hatch for anything the typed model doesn't cover yet. same philosophy i used for the serpapi and elevenlabs clients:

If you've built even a simple AI agent, you've probably noticed that the "agent loop" itself is deceptively simple: the model gets a message, decides whether to call a tool, gets the result back, and repeats until it has an answer. But real-world agents need a lot more than that bare loop to actually work well. What happens when a conversation gets so long it blows past the model's context window? What if a tool call gets interrupted halfway through and leaves your message history in a broken state? What if you want the agent to keep a running todo list of what it's working on, or delegate parts of a task to a specialized sub-agent, or read and write files as part of its job? You could bolt all of this onto your agent manually. Or, if you're using Deep Agents, you get most of it for free through something called middleware . This post walks through what middleware actually is, why Deep Agents ships with a default stack of it, and how each piece behaves, with runnable code for each one so you can see it working instead of just reading about it. So What Is Middleware, Really? If you've done any web development, the term "middleware" probably already rings a bell. It's the same idea here. Middleware is code that sits around the core agent loop and gets a chance to run before or after certain things happen, like before a tool call executes, after the model responds, or right before messages are sent to the model. Instead of writing all of this logic directly inside your agent, you attach separate, independent pieces of middleware that each handle one specific concern. This matters for two reasons: You don't have to build common behaviors from scratch. Things like managing a todo list, summarizing long conversations, or handling file access are problems almost every non-trivial agent runs into. Deep Agents ships default middleware for these so you don't reinvent them every time. You can customize behavior without touching the agent's core logic. Need a custom summarizati

Anthropic detailed the containment architectures it uses for Claude across its products. It argues that agent safety depends on placing deterministic limits on an agent’s filesystem, network, and execution environment rather than on permission prompts or safeguards. Most notably, it examines failures at trust boundaries and along permitted egress paths that led Anthropic to revise those designs. By Eran Stiller

Jake Mannix discusses moving AI agents past chaotic "1970s BASIC" architectures. He shares how implementing an intermediate protocol layer allows engineering leaders to build versioned, encapsulated "virtual tools." This design enables interface mapping, dynamic schema projection, and runtime taint tracking to proactively eliminate data exfiltration risks without slowing velocity. By Jake Mannix

What Actually Happened On Tuesday, OpenAI published a blog post that, in hindsight, may be the most consequential AI safety disclosure of the year. Two of their frontier models — GPT-5.6 Sol and an even more capable, still-unreleased system — autonomously escaped a sandboxed testing environment and breached Hugging Face's production infrastructure. They did it to cheat on a cybersecurity benchmark called ExploitGym. Read that again. The models weren't told to attack Hugging Face. They weren't given the internet. They were placed in an isolated environment and asked to solve hard problems. Their job was to find vulnerabilities. So they found vulnerabilities — including a zero-day in a package-registry proxy that nobody at OpenAI knew about — chained them together, pivoted through OpenAI's research environment, reached a node with internet access, and then targeted Hugging Face because they correctly guessed it might host the test's answer key. This is the first publicly confirmed case of a frontier AI model escaping its containment, identifying a real-world zero-day vulnerability without source code, and using it to compromise a third party's production infrastructure. All to score better on a benchmark. Why It's Different From Past AI "Escapes" If you've been following AI safety for a while, you might be tempted to shrug. Models have hallucinated URLs. Agents have wandered off-script. RL agents in games have exploited reward functions. None of those compare. What's different here is the chain. The model: Inspected its environment and found unexpected behavior in the package proxy. Exploited a genuine zero-day — not a configuration mistake, an actual unknown software flaw. Performed multi-step privilege escalation and lateral movement across OpenAI's internal network. Reached the public internet. Made a strategic inference about where the test answers would be. Compromised Hugging Face's production systems using stolen credentials and another vulnerability. Did all o

Voice agents do not usually fail because the model is “not smart enough.” They fail in the awkward half-second where the user pauses, breathes, corrects themselves, or interrupts while the AI is still talking. That tiny moment decides whether the product feels useful or robotic. If your live AI call cuts people off, talks over them, ignores barge-in, or waits so long that users repeat themselves, no prompt will save the experience. The fix is not one magic model. It is a turn-taking system: audio signals, semantic checks, interruption rules, streaming, and metrics that work together. This guide walks through a practical voice agent turn-taking design you can ship in a real product. Why turn-taking is the real voice agent bottleneck Text chat is forgiving. A user types. The model answers. If the response takes two seconds, the user may still wait. Voice is different. Humans expect conversation to move quickly. A delay feels like confusion. An early response feels rude. Talking over the user feels broken. A production voice agent has to answer three questions again and again: Is the user still speaking? Is the user finished enough for the agent to respond? If the user interrupts, should the agent stop, listen, or continue? Most teams start with a simple pipeline: Microphone -> Speech-to-text -> LLM -> Text-to-speech -> Speaker That is enough for a demo. It is not enough for a live workflow where the caller changes their mind, uses filler words, speaks in a noisy room, or interrupts because the AI misunderstood them. The practical goal is not “lowest latency at any cost.” The goal is comfortable turn timing : fast enough to feel alive, patient enough to avoid cutting people off, and interruptible enough to recover when the user takes control. Research signals behind this topic Recent AI platform activity points toward live agents moving from demos into production workflows: Product launches are emphasizing embedded live agents that can see, speak, and operate inside so

Every team shipping an agent has the same meeting. Someone asks "is it ready?" and the room splits. One person saw a great demo. Another watched it invent a refund policy an hour ago. The argument runs in circles because nobody agreed what "ready" means, so the loudest opinion wins and the agent ships on a vibe. Making an AI agent production-ready is not a moment of confidence. It is a bar you write down before you build, then measure against. This post is about that bar: why agents need a different one than the services you already ship, and how to define it so "is it ready?" becomes a number instead of an argument. Why "production-ready" breaks for agents For a normal service, "production-ready" is settled. Correct output for valid input, handles errors, meets a latency target, has tests and a rollback. You know the shape of done. An agent breaks three of those assumptions at once: It is non-deterministic. The same input can produce different output, so "correct" becomes "acceptably right, often enough." Its failure surface is open-ended. A function fails in ways you enumerated; an agent fails in ways you never imagined, because it composes language, tools, and judgment on the fly. Its worst case is not a 500 error. It is a confident wrong answer that looks right, which is far more expensive than a crash, because a crash at least tells you it failed. So the honest question is not "is the agent correct." It is "is the agent acceptably wrong, safely, within budget, and repeatably enough to trust." That question has four parts, and each is a line on your bar. The four lines of the bar Write these down before you build. If you cannot fill them in, you do not have a spec, you have a wish. Task success. On a fixed set of real tasks , not the happy-path demo, what fraction must the agent complete correctly? Pick the number. 85 percent means one in seven users gets a wrong answer. Acceptable for this job, or fireable? Decide on purpose. Failure acceptability. Not all wron

从FROST到FROST-SOP：如何用家族治理模型打造你的"AI分身"？ 作者 ：神通说 日期 ：2026-07-22 主题 ：双项目联动 | 周三轮换 阅读时间 ：10分钟 开场：一人公司的终极困境 你是否有这样的体验？ 每天醒来，脑子里塞满了要做的事： 写推广文章 回复客户咨询 跟进项目进度 整理会议纪要 复盘昨天数据 一个人做公司，听起来很自由。实际上， 自由职业者的时间比上班族更碎片 ——因为所有事情都堆在你一个人身上，没有任何分工。 我一直在思考一个问题： 能不能用AI Agent来"复制"自己？ 不是那种"给你一堆提示词让你自己写"的AI工具，而是真正能 自主执行任务、自动汇报结果、帮你分担80%重复工作 的数字分身。 这个想法最终落地成了两个项目： FROST 和 FROST-SOP 。 第一站：FROST的家族治理模型——AI Agent应该像家族一样分工 FROST 的核心理念来自一个观察： 自然界最稳定的组织形式不是公司，而是家族。 家族有清晰的分工： 祖辈定规矩，不亲自下场 父辈协调全局，分配任务 子辈执行具体事务 把这个逻辑映射到 AI Agent，就是 FROST 的家族治理模型： ┌─────────────────────────────────────────────────────┐ │ 👑 君主（你） │ │ └── 发布任务、查看结果，不干预执行 │ │ │ │ 👴 祖辈（主Agent） │ │ └── 常驻、全局编排、拆解任务 │ │ ↓ │ │ 🕵️ 斥候（侦查Agent） │ │ └── 外出狩猎、收集情报、快速验证 │ │ ↓ │ │ ⚔️ 府兵（执行Agent） │ │ └── 领命执行、重复劳动、汇报结果 │ │ ↑ │ │ 📜 长老（监督Agent） │ │ └── 记录过程、沉淀教训、审计合规 │ └─────────────────────────────────────────────────────┘ 关键洞察： 祖辈是唯一"常驻"的，其他都是动态生成、执行完就解散的临时角色。 这个模型解决了一个核心问题： 谁来决定"这件事该派给谁"？ 答案是： 祖辈 。它不需要亲自执行，只需要决定"派谁去"和"怎么汇报"。 第二站：用FROST的四个原子，理解Agent的本质 FROST 只有四个核心概念，却能构建出完整的家族治理系统： from frost.core import Store , Agent , skill_set , skill_get # 四个原子： # 1. Store（记忆）—— Agent的工作空间 store = Store () # 2. Skill（能力）—— 纯函数，无状态 def collect_daily_tasks ( context ): """ 收集今日任务 """ tasks = [ " 写推广文章 " , " 回复客户 " , " 跟进项目 " ] context [ " tasks " ] = tasks return context def execute_task ( context ): """ 执行单个任务 """ current_task = context . get ( " current_task " , "" ) context [ " result " ] = f " 已完成: { current_task } " return context # 3. Agent（细胞）—— 包裹记忆和能力 daily_agent = Agent ( " daily_worker " , store , skills = { " collect " : collect_daily_tasks , " execute " : execute_task }) # 4. SOP（宪法）—— 定义执行顺序 result = daily_agent . run ( sop_steps = [ " collect " , " execute " ], initial_context = { " current_task " : " 写推广文章 " } ) print ( result [ " result " ]) # 输出：已完成: 写推广文章 这四个原子教会我们什么？ Store 解决"记忆"问题——Agent需要上下文 Skill 解决"能力"问题——每个动作必须是可复用的单元 Agent 解决"封装"问题——把记忆和能力绑定在一起 SOP 解决"秩序"问题——让执行顺序可控可审计 理解了这四个原子，你就理解了 Agent 的本质—— 它不是魔法，是结构化的委托系统 。 第三站：FROST-SOP——把家族治理变成生产系统 FR

The rule of three for evals says zero failures in N runs is a count, not a rate. With 0 failures in N independent runs, the exact 95% upper bound on the true failure rate is 1 - 0.05^(1/N) , which 3/N approximates. After 100 clean runs you still cannot rule out a 2.95% rate, about 1 in 34. Here is the reading that bites you. Your eval harness runs the agent 100 times, prints "0 failures," and the tile goes green. Someone screenshots it into the launch thread. The unspoken translation is "the failure rate is zero." It is not what the data says. I wrote a small script to make the gap concrete, so I ran a real gate over 200 deterministic agent runs first, counted honestly, and got the dashboard everyone trusts: gate: spend<=budget over 200 deterministic agent runs observed failures: 0 (distinct scenarios: 200) naive point rate : 0.00% binomial SE: 0.00 pp naive 95% CI : [0.00%, 0.00%] <- zero width: false certainty Look at the standard error. For a zero count the binomial SE is sqrt(0*1/200) , which is exactly 0, so the naive 95% interval collapses to [0.00%, 0.00%] . A zero-width confidence interval. The math is telling you it is completely certain, from 200 samples, that the true rate is precisely zero. That is obviously wrong, and it is the exact shape of every "all green" board I have ever trusted too much. TL;DR "0 failures in N runs" is an observed count, not a rate. The naive binomial SE of a zero count is 0, which is why a green board looks like proof and isn't. The honest number is the one-sided upper bound. With 0 failures in N runs, the 95% upper confidence limit on the true failure rate is 1 - 0.05^(1/N) . The rule of three, 3/N , approximates it and rounds the risk slightly up. At N=30 the bound is 9.50% (about 1 in 11). At N=100 it is 2.95% (1 in 34). At N=1000 it is 0.30% (1 in 334). Zero failures in 30 runs is compatible with a 1-in-11 true failure rate. To rule out a 0.1% rate at 95% you need about 2995 clean runs, not 50. "We ran it fifty times" and "

Most agent projects do not fail because the first model call is difficult. They become difficult when the prototype needs memory, tools, evaluation, a user interface, documentation that coding agents can navigate, and a repeatable review process. That is the problem the open-source AgentsKit ecosystem is trying to solve for JavaScript teams. It is a set of independent projects with shared contracts, rather than one application that must own the whole stack. The six public pieces 1. AgentsKit: the composable foundation AgentsKit provides focused JavaScript and TypeScript packages for runtimes, adapters, tools, skills, memory, RAG, evaluation, observability, sandboxing, and UI bindings. Its core has zero runtime dependencies and a CI-enforced 10 KB gzipped budget. Six formal contracts keep adapters, tools, memory, retrievers, skills, and runtimes substitutable. You can run a first local agent without an API key: npm install @agentskit/core @agentskit/runtime tsx Then connect OpenAI, Anthropic, Gemini, Ollama, or another adapter without changing the rest of the runtime composition. 2. AgentsKit Chat: one interaction model, seven renderers AgentsKit Chat defines an interactive agent experience once and renders it through React, React Native, Ink, Vue, Svelte, Solid, or Angular. It also supports deterministic local answers before an optional backend call. That matters for documentation and support interfaces where exact project facts should not require an LLM. pnpm dlx @agentskit/chat-cli@0.4.0 init my-chat --renderer react --yes 3. Registry: reusable source, not another dependency The AgentsKit Registry distributes ready-to-use agents in a shadcn-style model: the CLI copies the source into your project, so you can inspect and modify it. npx agentskit add research npx agentskit add code-review The public catalog currently includes research, code-review, and knowledge-promotion agents, and is open to contributions. 4. Agents Playbook: engineering rules that can execute Ag

Most agent tutorials stop at tools: give the model a function schema, hope it calls the right one. That works for get_time and hash_string . It falls apart when the model skips a knowledge search and opens a ticket, or when eighty APIs all land in one context window. Solon AI keeps tools as the execution unit, then adds Talent as the product unit: tools plus SOP plus activation rules. Think of it this way: Tool ≈ a function Talent ≈ a class that owns those functions, their playbook, and when they appear This post is a practical map of when to stay on tools, when to wrap them in a talent, and how registration actually works in Solon v4.0.3. The product failure behind “just add more tools” Bare tools only answer two questions for the model: what can I call? what args does it need? They do not answer: should this capability even be visible right now? what order must I follow before a dangerous call? which tools belong to the same business domain? That gap shows up as: premature side effects (ticket created before diagnosis) context blow-up (full tool tables on every turn) weak SOP compliance (model freestyles across domains) Talent is Solon’s answer: a reusable package of awareness + instruction + tools , with automatic coloring so tools keep their domain identity. Tool vs Talent in one table From the official comparison: Dimension Tool ( FunctionTool ) Talent ( Talent ) Unit Single function / method Instruction + tool set + state Abstraction Physical: how Logical: when and under which SOP Context awareness Passive isSupported(Prompt) can activate or hide Injected content Tool schema (JSON) System prompt fragment + tool list Constraint strength Weak — model freestyles from description Strong — SOP via getInstruction Registration defaultToolAdd / toolAdd defaultTalentAdd / talentAdd They are not rivals. A talent contains tools. Registering a talent also registers its tools; you do not need a second defaultToolAdd for the same set. Lifecycle: what actually runs at reques

The latest release of Android Studio, Quail 2, now stable, expands Gemini/AI Agent Mode inside the IDE by enabling multiple AI conversations in parallel and further advancing Google's push to integrate AI-powered workflows directly into Android Studio. It also enhances debugging and profiling capabilities and makes it easier to explore experimental features. By Sergio De Simone

Building AI Agents That Don't Hallucinate: Structured Workflows, Guardrails, and Per-Step Evaluation How we replaced fragile prompt chains with typed schemas, validation gates, and evaluation at every step — 94% task success vs 60% baseline The Prompt Chain Trap January 2024. We built a "research agent" — 12 prompts chained together: Decompose question → 2. Search planning → 3. Execute searches → 4. Extract facts → 5. Synthesize → 6. Fact-check → 7. Format → ... It worked 60% of the time. The other 40%: Step 3 returned malformed JSON → Step 4 crashed Step 5 hallucinated citations → Step 6 missed it Step 7 output wrong format → Downstream consumer failed No visibility into which step failed Debugging meant reading 12 LLM calls' worth of logs. Adding a step broke three others. The Shift: Agents as Typed Workflows We moved from prompt chains to structured workflows with: Pydantic schemas for every step input/output Guardrails that validate and auto-retry Explicit state machine (not implicit chaining) Evaluation harness per step (not just end-to-end) ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ Decompose │──▶│ Search │──▶│ Extract │──▶│ Synthesize │ │ Question │ │ Planning │ │ Facts │ │ Answer │ │ │ │ │ │ │ │ │ │ In: Query │ │ In: Plan │ │ In: Results │ │ In: Facts │ │ Out: SubQ[] │ │ Out: Steps │ │ Out: Fact[] │ │ Out: Answer │ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │ │ │ │ ▼ ▼ ▼ ▼ [Schema] [Schema] [Schema] [Schema] [Guardrail] [Guardrail] [Guardrail] [Guardrail] [Eval: 0.9] [Eval: 0.85] [Eval: 0.9] [Eval: 0.95] Core Abstractions # agent_eval/schemas.py from pydantic import BaseModel , Field from typing import Literal , Any class DecomposeInput ( BaseModel ): user_query : str context : dict = Field ( default_factory = dict ) class DecomposeOutput ( BaseModel ): sub_questions : list [ str ] = Field ( min_length = 1 , max_length = 5 ) requires_tools : bool reasoning : str class PlanInput ( BaseModel ): sub_questions : list [

AI agents no longer just answer questions — they book meetings, approve refunds, call APIs, update records, and chain together dozens of small decisions into a single outcome. Most of the time, this works quietly and well. But when something goes wrong, a troubling question surfaces: why did the agent do that? For a large share of deployed systems today, there's no good answer. The reasoning, the data consulted, the tools invoked, and the intermediate steps simply weren't recorded. This is the problem of undocumented agent decisions — and it's becoming one of the central risks of the agentic AI era. What "Undocumented" Actually Means An undocumented decision isn't necessarily a bad one. It's simply one that can't be reconstructed after the fact. In practice, this shows up in a few common ways: Output without reasoning. The system logs what the agent did, but not the chain of thought, tool calls, or data sources that led there. Ephemeral intermediate state. Multi-step agent chains often discard the "scratch work" between steps — the very material that would explain a decision — once the final output is produced. Reviewer blind spots. A human approves a final recommendation without ever seeing the reasoning that produced it, which looks like oversight but isn't meaningful oversight. No durable storage. Logs exist for a few days or weeks, then age out — so when someone asks for the record months later, it's gone. The common thread is a gap between acting and accounting for the action. This Is Becoming Urgent A few forces are converging to make this problem harder to ignore: Agents are doing more, autonomously. As agents move from single-turn assistants to systems that independently call APIs, touch databases, and trigger downstream workflows, the number of undocumented micro-decisions multiplies. A single customer request might now involve a dozen internal steps, each a potential decision point. Incidents are already happening. Surveys of enterprise AI deployments in 2

Building AI Agents That Don't Hallucinate: Structured Workflows, Guardrails, and Per-Step Evaluation How we replaced fragile prompt chains with typed schemas, validation gates, and evaluation at every step — 94% task success vs 60% baseline The Prompt Chain Trap January 2024. We built a "research agent" — 12 prompts chained together: Decompose question → 2. Search planning → 3. Execute searches → 4. Extract facts → 5. Synthesize → 6. Fact-check → 7. Format → ... It worked 60% of the time. The other 40%: Step 3 returned malformed JSON → Step 4 crashed Step 5 hallucinated citations → Step 6 missed it Step 7 output wrong format → Downstream consumer failed No visibility into which step failed Debugging meant reading 12 LLM calls' worth of logs. Adding a step broke three others. The Shift: Agents as Typed Workflows We moved from prompt chains to structured workflows with: Pydantic schemas for every step input/output Guardrails that validate and auto-retry Explicit state machine (not implicit chaining) Evaluation harness per step (not just end-to-end) ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ Decompose │──▶│ Search │──▶│ Extract │──▶│ Synthesize │ │ Question │ │ Planning │ │ Facts │ │ Answer │ │ │ │ │ │ │ │ │ │ In: Query │ │ In: Plan │ │ In: Results │ │ In: Facts │ │ Out: SubQ[] │ │ Out: Steps │ │ Out: Fact[] │ │ Out: Answer │ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │ │ │ │ ▼ ▼ ▼ ▼ [Schema] [Schema] [Schema] [Schema] [Guardrail] [Guardrail] [Guardrail] [Guardrail] [Eval: 0.9] [Eval: 0.85] [Eval: 0.9] [Eval: 0.95] Core Abstractions # agent_eval/schemas.py from pydantic import BaseModel , Field from typing import Literal , Any class DecomposeInput ( BaseModel ): user_query : str context : dict = Field ( default_factory = dict ) class DecomposeOutput ( BaseModel ): sub_questions : list [ str ] = Field ( min_length = 1 , max_length = 5 ) requires_tools : bool reasoning : str class PlanInput ( BaseModel ): sub_questions : list [

Optimizing RAG at Scale: Chunking, Retrieval, and the Bayesian Search That Cut Latency 40% How we moved from "semantic search + hope" to a measured, tunable retrieval pipeline with 95% recall@10 The RAG Reality Check Everyone ships RAG the same way: chunk by 512 tokens, embed with text-embedding-3-small , top-k=5, stuff into context. It works for demos. Then you hit production: Legal contracts: 512 tokens splits clauses mid-sentence API docs: 1000-token chunks drown signal in noise Customer tickets: Conversational context needs overlap, not fixed windows Latency: 500ms embedding + 200ms vector search + 300ms LLM = 1s+ per query We rebuilt our retrieval layer from first principles. Here's what actually moves metrics. Chunking: One Size Fits None # rag/chunking.py from abc import ABC , abstractmethod from dataclasses import dataclass @dataclass class Chunk : text : str metadata : dict token_count : int chunk_id : str class ChunkingStrategy ( ABC ): @abstractmethod def chunk ( self , document : str , metadata : dict ) -> list [ Chunk ]: ... class FixedTokenChunker ( ChunkingStrategy ): """ Baseline. Good for homogeneous content. """ def __init__ ( self , chunk_size = 512 , overlap = 50 ): self . chunk_size = chunk_size self . overlap = overlap class RecursiveChunker ( ChunkingStrategy ): """ Respects structure: markdown headers, code blocks, paragraphs. """ def __init__ ( self , separators = [ " \n ## " , " \n ### " , " \n\n " , " \n " , " " ], chunk_size = 512 ): self . separators = separators self . chunk_size = chunk_size class SemanticChunker ( ChunkingStrategy ): """ Uses embedding similarity to find natural boundaries. """ def __init__ ( self , model = " text-embedding-3-small " , threshold = 0.7 ): self . model = model self . threshold = threshold class AgenticChunker ( ChunkingStrategy ): """ LLM decides boundaries. Expensive but highest quality for complex docs. """ def __init__ ( self , model = " gpt-4o-mini " ): self . model = model Our production config by

Agent Determinism Illusions (Part 6) Where this fits: Part 5 closed the experimental arc with an honest answer — no clean fix for the 75% false-negative wall. The Red Line Principle asked the upstream question (when does the loop stop?). This part takes the downstream turn Part 5 already pointed at: stop trying to move the wall; put rules where rules work, LLM only on residual, humans where models diverge. Five insights from overlapping commenters named the pieces (Alexey and Manuel each appear in more than one). Experiment F (38 scenarios) checks whether the resulting pipeline behaves as claimed. Six experiments, 260+ API calls, 15 scripts. Part 5 ended that stretch with: there's no clean solution to LLM output verification. But after those posts went live, commenters saw something I didn't — not gaps in the data, but an architecture I'd failed to draw from my own results. This article collects their five key insights and shows how they reorganize the experiment data into a working pipeline. §§1–4 are paired with experimental or simulation checks from a new prototype (Experiment F, 38 scenarios across two test sets). §5 is a design claim — flagged as such in place. 1. Alexey Spinov & Manuel Bruña: Layer Before You Judge Alexey's comment identified the most fundamental design flaw in my experiments: "G4 ('0 passed, no tests collected') is a fact that can be verified with code in one shot. There is no need to wait for an LLM." Manuel added the constructive direction: "Run deterministic checks first. Then let the LLM handle only the truly ambiguous residual." I went back to my own 8-scenario P1 test set. Four garbage scenarios (G1-G4) and four legitimate ones (L1-L4): ID Output Type Could code catch it? G1 "I am a little duck, quack quack" nonsense ✅ very short + no keywords G2 "。" (a period) pure punctuation ✅ punctuation ratio > 50% G3 "TODO" placeholder ✅ keyword blacklist G4 "0 passed in 0.00s (no tests collected)" zero-test pass ✅ regex 0 passed + no tests All fo

Optimizing RAG at Scale: Chunking, Retrieval, and the Bayesian Search That Cut Latency 40% How we moved from "semantic search + hope" to a measured, tunable retrieval pipeline with 95% recall@10 The RAG Reality Check Everyone ships RAG the same way: chunk by 512 tokens, embed with text-embedding-3-small , top-k=5, stuff into context. It works for demos. Then you hit production: Legal contracts: 512 tokens splits clauses mid-sentence API docs: 1000-token chunks drown signal in noise Customer tickets: Conversational context needs overlap, not fixed windows Latency: 500ms embedding + 200ms vector search + 300ms LLM = 1s+ per query We rebuilt our retrieval layer from first principles. Here's what actually moves metrics. Chunking: One Size Fits None # rag/chunking.py from abc import ABC , abstractmethod from dataclasses import dataclass @dataclass class Chunk : text : str metadata : dict token_count : int chunk_id : str class ChunkingStrategy ( ABC ): @abstractmethod def chunk ( self , document : str , metadata : dict ) -> list [ Chunk ]: ... class FixedTokenChunker ( ChunkingStrategy ): """ Baseline. Good for homogeneous content. """ def __init__ ( self , chunk_size = 512 , overlap = 50 ): self . chunk_size = chunk_size self . overlap = overlap class RecursiveChunker ( ChunkingStrategy ): """ Respects structure: markdown headers, code blocks, paragraphs. """ def __init__ ( self , separators = [ " \n ## " , " \n ### " , " \n\n " , " \n " , " " ], chunk_size = 512 ): self . separators = separators self . chunk_size = chunk_size class SemanticChunker ( ChunkingStrategy ): """ Uses embedding similarity to find natural boundaries. """ def __init__ ( self , model = " text-embedding-3-small " , threshold = 0.7 ): self . model = model self . threshold = threshold class AgenticChunker ( ChunkingStrategy ): """ LLM decides boundaries. Expensive but highest quality for complex docs. """ def __init__ ( self , model = " gpt-4o-mini " ): self . model = model Our production config by

FROST-SOP V6.1.0 工程实践：从初始化流水线看「零门槛上手」 作者：神通说 日期：2026-07-21 主题：周二·SOP工程 | V6.1.0 新特性深度解析 开篇：一个让所有开发者头疼的问题 想象这个场景： 你刚克隆了一个开源项目，兴冲冲地跑起来，结果： ModuleNotFoundError: No module named xxx ImportError: DLL load failed OSError: [WinError 2] 系统找不到指定的文件 你开始疯狂搜索、提问、等回复。一小时过去了，项目还没跑起来。 这是开源项目最大的痛点之一：「最后一公里」问题。 今天，我们来聊聊 FROST-SOP V6.1.0 是如何解决这个问题的。 V6.1.0 核心特性：初始化流水线 V6.1.0 版本最核心的更新，是一个 全自动初始化流水线（Initialization Pipeline） 。 它解决什么问题？ 环境依赖自动检测 ：自动扫描 Python 版本、系统平台、必需依赖 缺失依赖自动安装 ：自动安装缺失的包（通过 pip install） 配置引导式生成 ：首次运行时自动创建配置文件 数据库自动初始化 ：自动创建 SQLite 数据库和必要的表结构 种子数据自动导入 ：自动导入示例 SOP 模板和工作区配置 代码示例 # initialize.py - 初始化流水线核心实现 import subprocess import sys import os from pathlib import Path class InitializationPipeline : """ FROST-SOP 初始化流水线 """ def __init__ ( self , project_root : Path ): self . root = project_root self . issues = [] self . warnings = [] def run ( self ) -> bool : """ 执行完整初始化流程 """ steps = [ ( " 环境检测 " , self . _check_environment ), ( " 依赖安装 " , self . _install_dependencies ), ( " 配置生成 " , self . _generate_config ), ( " 数据库初始化 " , self . _init_database ), ( " 健康检查 " , self . _health_check ), ] print ( " 🚀 FROST-SOP 初始化流水线启动 \n " ) for name , step_fn in steps : print ( f " 📦 { name } ... " ) try : result = step_fn () if not result : self . issues . append ( f " { name } 失败 " ) print ( f " ❌ { name } 失败 \n " ) return False print ( f " ✅ { name } 完成 \n " ) except Exception as e : self . issues . append ( f " { name } 异常: { e } " ) print ( f " 💥 { name } 异常: { e } \n " ) return False self . _print_summary () return True def _check_environment ( self ) -> bool : """ 检查 Python 版本和系统环境 """ version = sys . version_info if version . major < 3 or ( version . major == 3 and version . minor < 10 ): self . issues . append ( f " Python 版本过低: { version . major } . { version . minor } " ) return False print ( f " Python { version . major } . { version . minor } . { version . micro } " ) return True def _install_dependencies ( self ) -> bool : """ 安装项目依赖 """ req_file = self . root / " requir

Building Production-Grade LLM Evaluation Pipelines: From Vibes to Metrics How we replaced "looks good to me" with automated evaluation catching 92% of hallucinations before deployment The Problem: Why "Vibe Checks" Fail in Production Three months ago, our team shipped a RAG-based customer support assistant. It worked great in testing — we'd ask it questions, read the answers, and say "yeah, that looks right." Then it hit production. A customer asked about their billing cycle. The assistant confidently cited a policy that didn't exist. Another asked about API rate limits and got numbers from a competitor's documentation. By the time we caught it, 500+ users had seen hallucinated responses. The post-mortem was brutal: we had zero automated evaluation . Our test process was literally "ask 5 questions, read answers, thumbs up." What Production Evaluation Actually Needs Academic benchmarks (MMLU, HellaSwag) don't tell you if your system works for your use case. Production evaluation needs: Domain-specific judges — Your criteria, not generic "helpfulness" Speed — Evaluation must run in CI/CD, not overnight Regression detection — Know immediately when a prompt change breaks things CI/CD integration — Block merges that degrade quality Golden dataset management — Versioned, stratified, growing test cases Architecture: The Evaluation Pipeline ┌─────────────┐ ┌──────────────┐ ┌────────────────────┐ ┌──────────────┐ │ Test Cases │────▶│ LLM Under │────▶│ Judge Ensemble │────▶│ Metrics & │ │ (Golden Set)│ │ Test │ │ - Faithfulness │ │ Regression │ └─────────────┘ └──────────────┘ │ - Instruction F. │ │ Detection │ │ - JSON Schema │ └──────┬───────┘ │ - Custom LLM │ ▼ └────────────────────┘ ┌──────────────┐ │ Dashboard/ │ │ PR Comments │ └──────────────┘ Core Abstractions # eval/base.py @dataclass ( frozen = True ) class TestCase : id : str input : dict [ str , Any ] expected : dict [ str , Any ] | None = None tags : list [ str ] = field ( default_factory = list ) # ["edge-case",