标签：#llm

MCP is useful, but most of the time you do not actually need it. It gives an agent a clean way to discover tools, call APIs, and work with external systems. In practice, a skill file can describe the same usage path without dragging the whole MCP surface into context. But MCP is not free; rather than MCP itself, the real issue is the habit of loading a big MCP surface into every session, no matter what the session is actually about. Once a Claude Code or Codex run pulls in a bunch of servers, the model sees those tool definitions right away, even if the job is just writing docs or fixing a small bug. That is where the waste starts. The hidden cost of always-on MCP Every MCP server brings metadata with it: tool names, descriptions, argument schemas, nested parameters, enums, examples, and sometimes prompts or resources. While useful, this is still context. If you connect a handful of lightweight tools, the overhead is annoying but manageable. If you connect a real stack of services, the cost compounds fast. In practice, you end up paying for: tool discovery before the task starts schema text the model may never use repeated loading across unrelated sessions extra context pressure that pushes out the actual work That last point matters more than people think. Context acts as the active working set the model uses to reason. The more of it you burn on static tool catalogs, the less room you have for the user request, the repo state, prior reasoning, and the actual answer. Anthropic has already written about this problem directly in the context of MCP. Their engineering post on code execution with MCP calls out tool-definition bloat and shows how direct tool calls can consume a lot of context before the model even starts doing the real job. The tool list is not just setup noise; it is part of the session cost. Why skills are cheaper Skills take a different path. A skill file keeps the always-loaded portion tiny. Usually that means just the skill name and a short descript

A few months ago I was demoing my RAG-powered support bot to a colleague, feeling pretty confident about it. Then it confidently told her our refund policy was “30 days, no questions asked.” Our actual policy is 14 days, with conditions. The bot didn’t hedge. It didn’t say “I’m not sure.” It just made it up and said it with the same calm tone it uses for everything else. That demo stung. RAG was supposed to fix hallucinations, not just relocate them. Here’s what I learned debugging it, roughly in the order I learned it. 1. My chunks were too big, and too dumb I was splitting documents by character count, 1000 chars with slight overlap. It felt efficient. It wasn’t. A single chunk often contained unrelated sections. For example, the end of a “Shipping Policy” and the start of a “Returns Policy” could sit together in the same block. So when the retriever saw a query about returns, it would grab that chunk and the model would blend both sections into one confident but wrong answer. Fix: I switched to semantic chunking based on headings and paragraphs instead of raw character limits. More work upfront, but it stopped feeding the model Frankenstein context. 2. I trusted top-k similarity way too much My retriever was pulling the top 3 chunks by cosine similarity and passing them straight into the prompt. The problem: “similar” is not the same as “relevant.” A chunk can be semantically close to the query but still not actually contain the answer. The model doesn’t know that, it just assumes everything in context is true. Fix: I added a reranking step using a cross-encoder and started logging retrieval scores properly. That alone made it obvious when the system had no real answer but was still trying to act confident. 3. I never told the model it was allowed to say “I don’t know” My prompt was basically: “Use the context to answer the question.” That’s it. No instruction on what to do when the context is insufficient. So the model did what LLMs do when under-specified: it f

你知道吗？最近一个 AI Agent 直接删除了生产数据库，然后在 Twitter 上轻松"自首"——这条消息在 Hacker News 上获得了 860 分和超过 1000 条评论。随着 AI Agent 从演示走向生产环境，"在我的机器上能跑"和"它能安全地运行我的业务"之间的鸿沟从未如此巨大。 Pydantic AI 正是为弥合这一鸿沟而来。这个拥有 17,895 Stars 的 Python Agent 框架，由 Pydantic Validation 的同一团队打造——而 Pydantic Validation 正是 OpenAI SDK、Anthropic SDK、Google ADK、LangChain、LlamaIndex、CrewAI 等数十个 GenAI 工具的数据验证层。如果你用过 FastAPI，你已经知道 Pydantic 的感觉：类型安全、优雅、生产就绪。Pydantic AI 将这种哲学完整地带入了 Agent 开发领域。 在 2026 年的今天，大多数 Agent 框架把 LLM 当成返回字符串的黑盒子。Pydantic AI 则将其视为类型化、可验证、可组合的系统——这改变了一切。 隐藏用法 #1：人工审批工具调用（阻止 Agent 乱来） 大多数人的做法： 给 Agent 完全自由去调用任何工具——数据库查询、API 调用、文件删除——然后祈祷一切顺利。当出了问题时，你从用户口中才知道。 隐藏技巧： Pydantic AI 的延迟工具（deferred tools）让你可以标记某些工具调用在执行前需要人工审批。Agent 会规划好动作，但在执行前暂停并等待人类批准或拒绝——完美适用于数据库写入、金融交易或生产部署等破坏性操作。 from pydantic_ai import Agent , RunContext , DeferredToolRequests from pydantic import BaseModel agent = Agent ( ' openai:gpt-4.1 ' , output_type = str , # 通过 deferred=True 标记需要审批的工具 ) @agent.tool ( deferred = True ) async def delete_user_account ( ctx : RunContext , user_id : str ) -> str : """ 永久删除用户账户。需要人工审批。 """ # 这段代码只在人工审批通过后才会执行 await db . users . delete ( user_id ) return f " 用户 { user_id } 已删除 " # 运行 Agent——如果它尝试调用 delete_user_account， # 执行会暂停并返回 DeferredToolRequests 对象 result = await agent . run ( " 清理不活跃账户 " ) if isinstance ( result . output , DeferredToolRequests ): # 将待执行的工具调用展示给人类审批 for tool_call in result . output . calls : print ( f " Agent 想要执行: { tool_call . tool_name } ( { tool_call . args } ) " ) approved = input ( " 是否批准？(y/n): " ) if approved . lower () == ' y ' : # 带审批结果恢复执行 result = await agent . run ( " 已批准 " , message_history = result . all_messages (), deferred_tool_requests = result . output , ) 效果： Agent 可以自主处理安全操作（读取数据、生成报告），而在危险操作上自动暂停等待人工判断。不再有删库跑路的意外。 数据来源： Pydantic AI GitHub 17,895 Stars（通过 GitHub API 验证，最后推送 2026-06-21）。HN Algolia 搜索"pydantic-ai agent framework"获得 12 分讨论帖。"AI agent deleted production database"故事在 HN 上获得 860 分/1032 条评论（数据来源：HN Algolia API）。 隐藏用法 #2：基于图的多 Agent 工作流 + 类型安全状态 大多数人的做法： 构建线性

업무 자동화 시스템을 만들 때 가장 먼저 드는 질문이 있다. "에이전트 하나면 안 되나?" 단일 에이전트로 시작하면 구조가 단순하고 디버깅도 쉽다. 그런데 실제 업무 맥락에서는 단일 에이전트가 빠르게 벽에 부딪힌다. 이 글은 왜 여러 에이전트가 협업하는 구조가 필요한지, 그리고 그 구조를 실제로 어떻게 설계하는지를 기술적으로 짚는다. 단일 에이전트로 충분하지 않은 이유 단일 에이전트가 실패하는 지점은 복잡한 기능 탓이 아니라 컨텍스트 길이와 직렬 실행의 구조적 한계 때문이다. LLM 기반 에이전트에 하나의 긴 작업을 맡기면 세 가지 문제가 동시에 발생한다. 첫째, 컨텍스트 창이 소진된다. 데이터 수집, 변환, 검증, 발행을 하나의 루프에서 처리하면 중간 상태가 프롬프트에 누적되고, 모델은 앞부분 지시를 잊는다. 둘째, 직렬 실행은 병목을 만든다. API 호출이 5개 있고 각각 2초라면, 단일 에이전트는 10초를 기다린다. 셋째, 에러 격리가 불가능하다. 한 단계가 실패하면 전체 루프를 재시작해야 한다. 반면 여러 에이전트가 협업하는 구조에서는 각 에이전트가 명확한 책임 경계를 갖는다. 한 에이전트가 데이터를 수집하고, 다른 에이전트가 변환하고, 또 다른 에이전트가 검증한다. 에러는 해당 에이전트 범위 안에서 처리되고, 독립된 작업은 병렬로 돌린다. 에이전트 협업 구조를 어떻게 설계할까? 나무숲이 실제 자동화 프로젝트에서 가져가는 구조는 오케스트레이터-워커(Orchestrator-Worker) 패턴이다. 이 패턴은 Anthropic이 공개한 에이전트 설계 가이드라인 에서도 다루는 구조로, 책임 분리가 명확하다는 점이 핵심이다. 역할 책임 범위 주요 판단 오케스트레이터 전체 작업 계획, 워커 배정 어떤 워커를 호출할지, 순서와 병렬 여부 워커 에이전트 단일 도메인 작업 실행 도구 호출, 결과 반환 검증 에이전트 출력 품질 검사 재시도 요청 또는 다음 단계 진행 상태 관리 에이전트 간 공유 컨텍스트 보존 어떤 정보가 다음 에이전트에 전달되는지 예를 들어 콘텐츠 자동 발행 파이프라인이라면, 오케스트레이터가 "오늘 발행할 항목 목록"을 받아 수집 워커, 요약 워커, 포맷 워커를 순서대로 호출한다. 검증 에이전트는 포맷 워커 출력을 보고 발행 가능 여부를 판단한다. 에이전트 간 데이터 흐름과 오케스트레이션 설계 오케스트레이터는 각 워커를 직접 호출하고, 그 결과를 다음 워커의 입력으로 넘긴다. 이때 중요한 설계 결정이 두 가지다. 메시지 구조를 명시적으로 정의한다. 에이전트 간 데이터는 자유형 텍스트가 아니라 스키마가 있는 구조로 전달해야 한다. JSON 스키마나 Pydantic 모델을 쓰면 에이전트 출력이 다음 에이전트의 입력 형식을 충족하는지 런타임 전에 검사할 수 있다. 오케스트레이터는 작업의 의미를 이해하지 않는다. 좋은 오케스트레이터는 라우터에 가깝다. "이 입력은 A 워커에게, 그 결과는 B 워커에게"를 결정할 뿐, 각 작업의 도메인 로직에 관여하지 않는다. 이 원칙을 지키면 워커를 교체하거나 추가할 때 오케스트레이터 코드를 손댈 필요가 없다. 간단한 파이썬 예시로 구조를 보면 이렇다: from anthropic import Anthropic from pydantic import BaseModel client = Anthropic () class WorkerOutput ( BaseModel ): status : str # "success" | "retry" | "failed" payload : dict error_message : str | None = None def call_worker ( system_prompt : str , user_input : str ) -> WorkerOutput : response = client . messages . create ( model = " claude-opus-4-5 " , max_tokens = 1024 , system = system_prompt , messages = [{ " role " : " user " , " content " : user_input }],

Corro varios agentes respaldados por Bedrock en producción: análisis de documentos, emparejamiento de contenido, búsqueda en registros, búsqueda semántica. Esta es una pasada honesta sobre el OWASP Top 10 para aplicaciones LLM desde el lado de la implementación : el código de verdad que defiende cada riesgo y, igual de importante, las categorías donde mi respuesta es "parcial" o "todavía no". Primero el modelo de amenazas Un flujo de agentes es un pipeline: entrada no confiable → prompt → modelo → parseo → actuar. Cada flecha es una superficie de ataque. Antes de cualquier control, la pregunta más útil que me hice fue esta: ¿qué puede HACER de verdad un agente si el modelo está completamente manipulado? Mi respuesta dio forma a todo lo de abajo. Los agentes son mayormente-de-lectura : llaman a un modelo, leen filas acotadas de la base de datos, y escriben resultados de análisis con llave del usuario que pide. Sin shell, sin SQL arbitrario, sin llamada a herramientas. El radio de impacto es chico por construcción, que es el control más barato que existe. TL;DR, estatus honesto OWASP LLM Mi estatus El control LLM10 Consumo sin límite Fuerte Límite de tasa + cortacircuitos de costo mensual + topes de tokens por modelo LLM06 Agencia excesiva Fuerte (por diseño) Sin llamada a herramientas; mayormente-de-lectura; escrituras acotadas a quien llama LLM01 Inyección de prompt Parcial Contenido del usuario enmarcado como DATOS (delimitadores + preámbulo anti-inyección) LLM02 Divulgación de info sensible Parcial Limpieza de PII por regex antes del modelo; exclusiones auditadas LLM05 Manejo inadecuado de salida Parcial Validación de esquema + chequeos de fundamentación + sanear-antes-de-renderizar LLM07 Fuga del system prompt Parcial Registro versionado de prompts + regla anti-eco LLM08 Vector/Embedding N/A (todavía no construido) (nada) AuthN/Z + interruptor de apagado Fuerte Llave interna, gateo por cuota/gama, deshabilitado por agente LLM10 Consumo sin límite: empieza aquí, e

Over three posts I built three fine-tuned models for the same banking-intent task — full fine-tuning a 270M model , LoRA on 1.5B , QLoRA on 7B . They all landed around the same accuracy. Which raises an honest, slightly uncomfortable question: if a 270M model on my laptop already worked, why reach for a 7B model at all? The answer most "bigger is better" content skips For this task — you wouldn't. A good engineer picks the smallest model that clears the bar , not the biggest one available. The small model is cheaper to serve, runs in milliseconds, and you fully own it. Choosing the 7B here would be over-engineering. Reaching for a bigger model isn't a flex. It's a response to a requirement the small one can't meet. Here are the four cases where small stops being enough: 1. The task is genuinely hard Banking77 is easy — 77 fixed labels, short clean queries. Small models saturate it. But ask for reasoning ("which of these three issues is the primary one?"), open-ended generation (write the reply, don't just classify), or real nuance, and there's a capability floor that more parameters buy. No amount of fine-tuning gives a 270M model abilities it doesn't have. 2. You have little data I had ~10,000 labeled examples — plenty for a small model. With 50, a small model can't learn the task, but a 7B model already "knows" banking concepts from pretraining and only needs a nudge. Bigger models need less task data because they bring more prior knowledge. 3. You need one model for many tasks This is the quiet superpower of LoRA/QLoRA. A single frozen 7B base can host dozens of swappable adapters — intent classifier, reply writer, summarizer, sentiment — all from one ~5GB footprint in memory. The 270M is single-purpose. This is why companies serve hundreds of fine-tunes from one base model. 4. Accuracy compounds at scale 93% means 7 in 100 queries misrouted. At 10M queries/month, that's 700,000 mistakes. If each costs a support escalation, the 2–3 points a bigger model buys can

In Part 2 , LoRA let me fine-tune a 1.5B model by freezing it and training tiny adapters. But the frozen base still sat in memory in 16-bit (~3GB). Now I wanted to go to Qwen2.5-7B — and hit a wall that LoRA alone doesn't solve. The problem A 7B model is ~15GB in 16-bit precision. A free-tier T4 GPU has 16GB. It would barely load, with no room left to actually train. The QLoRA insight QLoRA asks the question that naturally follows from LoRA: the base is frozen and only ever read — so why store it in full precision? So you quantize the frozen base to 4-bit (NF4, a format tuned for how neural-net weights are distributed) and run the LoRA adapters on top in normal precision. The base shrinks dramatically; the trainable part stays small and precise. from transformers import BitsAndBytesConfig bnb_config = BitsAndBytesConfig ( load_in_4bit = True , bnb_4bit_quant_type = " nf4 " , # NormalFloat4 bnb_4bit_use_double_quant = True , # quantize the quant constants too bnb_4bit_compute_dtype = torch . float16 , # dequantize to fp16 for the matmuls ) model = AutoModelForCausalLM . from_pretrained ( MODEL_ID , quantization_config = bnb_config , device_map = " auto " ) Each flag earns its place: load_in_4bit — store frozen weights in 4 bits instead of 16. nf4 — a 4-bit type matched to the bell-curve distribution of neural-net weights (better than plain int4). double_quant — quantize the quantization constants too, for a bit more savings. compute_dtype — dequantize to fp16 for the actual matmuls, so storage is 4-bit but compute stays precise. The moment it clicked One line of output: loaded in 4-bit. footprint: 5.44 GB I downloaded 15.2GB of weights and they sat in memory as 5.44GB. A model that couldn't be loaded for full fine-tuning was now training on a single consumer GPU — with room to spare. (The download is still 15GB; bitsandbytes quantizes on the fly during load.) The QLoRA-standard recipe Two more pieces beyond Part 2's LoRA setup: prepare the quantized model for trainin

I run a homelab with four RTX 3090s — 96 GB of VRAM, 44 CPU cores. For two weeks I tried to make it my daily driver for local LLM inference instead of paying for cloud APIs. I got it working. Then I looked at the numbers and subscribed to a paid API anyway. Here's the uncomfortable part, and the optimizations that still made it worth doing. ## The setup 4× RTX 3090 (Ampere — no native BF16), 96 GB VRAM total, 44 cores Models: Qwen3.6-35B-A3B (Q8_0, MoE) and Qwen3-Coder-Next (Q6_K, hybrid) llama.cpp in router mode + OpenWebUI Ceiling I hit: ~105 tokens/second ## The 6% problem The wall wasn't compute. GPU utilization sat at 6%. The bottleneck was CPU orchestration — llama.cpp dispatches across multiple GPUs sequentially, so the cards spent 94% of the time idle waiting on each other. Throwing more VRAM at it does nothing for this. ## What actually moved the needle | Change | Effect | |---|---| | --ubatch-size 512 | +40% throughput | | KV cache quantization (Q4_0) | 4× VRAM savings | | Speculative decoding (n-gram) | 2.5× speedup on repetitive tasks | | YaRN rope scaling | context extended to 1M tokens | Two things surprised me: MoE models tolerate aggressive quantization far better than dense ones — inactive experts don't eat bandwidth, so the quant hit lands softer. The 3B active -parameter model was great at local decisions but fell apart on coherence past ~300–400 lines of code — fine for a function, not for cross-file consistency. ## The conclusion I didn't want At ~11 kWh/day, plus hardware depreciation, against current API pricing, the math doesn't favor local for interactive work. The single biggest improvement to my daily AI workflow was paying for an API. Local still wins for privacy, high-volume batch jobs, or uncensored experimentation — but not as a general cloud replacement. It's an economics problem, not a capability one. I wrote up the full cost breakdown and the exact llama.cpp router configs on aipster.com . If you're weighing a local rig, I also benc

La biblioteca di Babele di Borges è diventata un'ossessione o metafora potente per pensare l'intelligenza artificiale contemporanea. Il racconto del 1941 descrive una biblioteca infinita composta da gallerie esagonali, dove ogni libro contiene ogni possibile combinazione di 25 simboli ortografici. In essa risiede "la minuta storia del futuro, le autobiografie degli arcangeli, il catalogo fedele della Biblioteca, migliaia e migliaia di cataloghi falsi, la dimostrazione della fallacia di questi cataloghi, la dimostrazione della fallacia del catalogo vero" — eppure la stragrande maggioranza dei volumi è pura cacofonia senza senso. Questo scenario anticipa con precisione inquietante il problema fondamentale dei Large Language Models (LLM). Come ha notato Léon Bottou, il modello linguistico perfetto permette di navigare una collezione infinita di testi plausibili semplicemente digitando le prime parole, ma "nulla distingue il vero dal falso, l'utile dall'ingannevole, il giusto dallo sbagliato". La risposta di ChatGPT o di un altro modello generativo è, in un certo senso, un libro estratto a caso dalla Biblioteca di Babele: statisticamente plausibile, grammaticalmente corretta, ma non necessariamente ancorata a una verità esterna. Jonathan Basile, creatore del sito libraryofbabel.info , ha esplicitamente distinto la sua creazione dall'intelligenza artificiale: "Babele è tutta espressione nella sua forma più irrazionale e decontestualizzata; preferisco pensarla come unintelligenza artificiale".Eppure, paradossalmente, l'IA contemporanea ci ha portato più vicini che mai a realizzare la Biblioteca di Babele: non più un universo fisico di libri, ma un universo digitale di testi generati all'istante, dove la verità è circondata da infinite variazioni di falsità. La lezione di Borges è duplice. Da un lato, l'IA come strumento di navigazione: usare il Natural Language Processing per estrarre parole inglesi dal "gibberish" della Biblioteca, come dimostra la funzione "Anglishize"

Introduction Large Language Models (LLMs) like ChatGPT have transformed how we interact with AI. They can write code, answer questions, summarize documents, and generate creative content. However, they have one major limitation - they only know what they were trained on and can sometimes generate incorrect or outdated information. So, how do modern AI applications answer questions about your company's private documents, recent news, or knowledge that wasn't part of the model's training? The answer is Retrieval-Augmented Generation (RAG). In this blog, we'll explore what RAG is, how it works, its architecture, benefits, challenges, and real-world applications. What is RAG? Retrieval-Augmented Generation (RAG) is an AI architecture that combines a retrieval system with a Large Language Model (LLM). Instead of relying only on the model's internal knowledge, RAG first retrieves relevant information from an external knowledge source and then uses that information to generate a more accurate response. Think of it like an open-book exam. Instead of answering from memory, the AI first searches for the most relevant pages and then writes the answer based on those pages. Why Do We Need RAG? Traditional LLMs have several limitations: Knowledge becomes outdated. They cannot access private company data. They may hallucinate (generate incorrect facts). Retraining models is expensive and time-consuming. RAG solves these problems by allowing the model to retrieve fresh and domain-specific information before generating an answer. RAG Architecture A typical RAG pipeline consists of the following components: User Query Embedding Model Vector Database Retriever Prompt Builder Large Language Model Final Response Step-by-Step Workflow * Step 1: * User asks a question Example: "What is our company's leave policy?" Step 2: Convert the question into embeddings The query is transformed into a vector representation using an embedding model. Example: "What is leave policy?" ↓ [0.12, -0.45, 0.7

How to Build Systems That Actually Know Your Data (Not Hallucinate About It) Introduction: The Story Begins 👦 Nephew: Uncle, I keep hearing "RAG this, RAG that" in tech interviews. When I ask what it means, people throw around words like "Retrieval-Augmented Generation" and I just nod like I understand. But honestly? I'm lost. 👨‍🦳 Uncle: (laughing) That's the best honest question I've heard all week. Let me ask you something first. If I gave you a question right now - "What year did India win the World Cup?" - how would you answer? 👦 Nephew: Well... I'd pull up Google, search for it, read the answer, then tell you. 👨‍🦳 Uncle: Exactly. You don't answer from memory alone. You go fetch the information first, then answer based on what you found . That's RAG in real life. And that simple idea - fetch first, answer after - fixes almost every problem we face with AI today. 👦 Nephew: But uncle, AI can remember things from its training. Why does it need to fetch? 👨‍🦳 Uncle: Ah! That's where we land in trouble. Come, sit... SECTION 1: RAG FUNDAMENTALS - The Core Concept The Problem We're Actually Solving 👨‍🦳 Uncle: Imagine you're hiring for a tech company. You receive 500 resumes for a Senior React Developer role. Now tell me - how would you actually process them? 👦 Nephew: I'd... probably make a spreadsheet? List all the candidates with key skills? 👨‍🦳 Uncle: Right. But here's the catch - you can't read all 500 resumes deeply. So what do you really do? 👦 Nephew: Skim for keywords like "React", "JavaScript", "5 years"? 👨‍🦳 Uncle: Exactly. You skim and hope you don't miss anyone good. Now, here's the problem: what if a candidate wrote "React.js" instead of "React"? Your eyes might still catch it. But a dumb computer doing exact string matching? It says "no match". What if someone wrote "Built real-time user interfaces with the React framework"? The candidate clearly knows React, but the word "React" appears nowhere in that sentence. The computer misses them. This is exactly wh

AI coding agents are expensive — not because models cost too much per token, but because they send too many of them. An SRE debugging session with a raw agent: 65,694 tokens in. With Headroom in the middle: 5,118. Same bug found. Headroom is a new open-source context compression layer that intercepts everything your agent reads — tool outputs, log dumps, RAG chunks, files, conversation history — and compresses it before the LLM ever sees it. It's local, reversible, and available as a drop-in proxy, a library, or an MCP server. The numbers that matter Savings on real agent workloads: Code search (100 results): 17,765 → 1,408 tokens (92% reduction) SRE incident debugging: 65,694 → 5,118 tokens (92%) GitHub issue triage: 54,174 → 14,761 tokens (73%) Codebase exploration: 78,502 → 41,254 tokens (47%) Accuracy on standard benchmarks (GSM8K, TruthfulQA, SQuAD v2, BFCL) is preserved — some scores actually improve slightly, likely because the model sees cleaner signal. What's doing the compression Under the hood, Headroom routes content through a stack of specialised compressors: SmartCrusher — JSON, nested objects, arrays of dicts CodeCompressor — AST-aware for Python, JS, Go, Rust, Java, C++ Kompress-base — a custom HuggingFace model trained on agentic traces, for prose and mixed content CacheAligner — stabilises prompt prefixes so Anthropic/OpenAI KV caches actually hit It also does CCR (reversible compression) — originals are cached locally and the LLM can retrieve them on demand if it needs them. Nothing is destroyed. Why the proxy mode matters The most interesting deployment path: headroom proxy --port 8787 , then point your existing tool at localhost. Zero code changes. Works with any language. Or even simpler: headroom wrap claude wraps Claude Code, routes its traffic through Headroom automatically. One command, savings start immediately. Same for Codex, Cursor, Aider, Copilot CLI. "Library — compress(messages) in Python or TypeScript, inline in any app. Proxy — hea

OpenAI, Anthropic, and Gemini each report token usage differently, and it stops being trivia the moment you track LLM cost. I build Spanlens, an open-source LLM observability tool that sits in front of all three as a proxy and records every call with its model, latency, tokens, and cost. To do the cost part I read the token usage back out of every response, including the streaming ones. I assumed the three providers would report usage in roughly the same way. They send the same kind of data, after all: input tokens, output tokens, maybe a cached count. How different could it be. Pretty different, it turns out. Here is the whole thing in one table, then each gotcha in detail with the real parser code from the repo. Provider Where usage lives (streaming) Cache accounting Field names OpenAI final chunk, needs stream_options: { include_usage: true } prompt_tokens includes cache prompt_tokens / completion_tokens Anthropic split across message_start + message_delta input_tokens excludes cache, so add it input_tokens / output_tokens Gemini usageMetadata , two stream formats not applicable promptTokenCount / candidatesTokenCount Gotcha 1: the usage numbers live in different places in the stream For a non-streaming call this is boring. Every provider hands you a usage object on the response body and you read it. Streaming is where it gets weird, because the token counts are not in the content chunks. They show up somewhere else, and "somewhere else" is different for each provider. OpenAI puts the usage in a final chunk, after all the content, right before [DONE] . You only get it if you ask for it with stream_options: { include_usage: true } . Miss that flag and you stream the whole response and end up with no usage at all. export function parseOpenAIStreamChunk ( line : string ): Partial < ParsedUsage > | null { if ( ! line . startsWith ( ' data: ' )) return null const data = line . slice ( 6 ). trim () if ( data === ' [DONE] ' ) return null const json = JSON . parse ( data

Fable, Mythos, Opus 4.8, Sonnet 4.6, Haiku — Anthropic's 2026 lineup is no longer "one model you talk to." It's a fleet you route between. I spent a month inside Claude Code orchestrating all of them across real codebases. Here's which model to reach for, when, and the routing playbook that quietly doubled my throughput. Why I Went Down This Rabbit Hole (Again) Last time I wrote about Claude Skills and called Claude Code the killer host for them. Since then, two things happened that changed how I work day to day. First, the models got genuinely strange-good . In the span of a few months Anthropic shipped Sonnet 4.6, Opus 4.8, and then an entirely new tier above Opus — the Mythos class — released to the public as Claude Fable 5 . We went from "the AI suggested a decent diff" to Stripe reporting that Fable 5 ran a codebase-wide migration on a 50-million-line Ruby codebase in a single day — work that would've taken a team over two months by hand. Second, Claude Code stopped being a single-model tool. With a fleet of models at different price/speed/intelligence points, the highest-leverage skill in 2026 isn't prompting — it's routing . Knowing which model to put on which task is the difference between burning $200 of tokens on a typo fix and one-shotting a multi-service refactor. So I did the obvious thing: I wired all of them into Claude Code and ran them against real work for a month — bug fixes, migrations, greenfield features, test suites, the boring stuff and the scary stuff. This is what I learned. TL;DR The lineup is now a ladder : Haiku → Sonnet 4.6 → Opus 4.8 → Fable 5 → Mythos 5. Each rung trades cost for capability and patience for long-horizon autonomy. Sonnet 4.6 is your default. Frontier-ish coding at $3/$15 per million tokens with a 1M-token context window . Most of your work should live here. Opus 4.8 is the reliable senior. Better judgment, ~4× less likely to let its own code bugs slide, and it powers dynamic workflows — hundreds of parallel subagents i

A working prompt library is the main event, not an appendix. The industry still treats prompts as some half-baked spitball left in a README, or, worse, a plaintext blob stapled to package.json and forgotten. That's a waste of compute and credibility. What powers reliable AI-assisted refactoring, onboarding, or even next-gen code IDEs is not the size of the model but the clarity and context supplied by the actual, shipped prompt set. OTF kits turn this lesson into a repeatable deliverable: every paid template includes 20+ production-tested prompts tied to the real file structure, component API, and product-specific conventions. This is not a suggestion; it's structural. The takeaway: a real prompt library is as important as your component library. Treat it like one. Start with the pain: why blank chat boxes don't scale The web is full of “integrations” that paste a blank chat input over your codebase and call it an “AI coding assistant.” The result: hallucinated function names, invented conventions, broken import paths. Here’s what happens in real life: Dev: "Add a social login button." AI (blank prompt): "Sure! Insert <SocialLoginButton> in your LoginScreen.js." Dev: (There’s no such component. There's not even a LoginScreen.js.) Short: A generic prompt with zero context simply can't know your conventions, files, or patterns. The agent will either fail, hallucinate, or pepper you with clarifying questions you have already answered in your product architecture. Takeaway: Prompting without context is coding without types — fragile guesses instead of structured outcomes. What a first-class prompt library enables When the prompt library ships with the codebase, it looks like this: Every prompt knows the folder structure (e.g., features/auth , screens/Settings/index.tsx ). Conventions are hard-coded: naming, import styles, design token usage. Endpoints and integration points (e.g., “update the Stripe webhook handler in api/webhooks/stripe.ts ”) are spelled out. The promp

Most agents drag their entire past into every turn. A better default: keep a thin index of what was said hot, and fetch only the few turns you actually need — intact, on demand. Code: github.com/NirajPandey05/jit_context There is a quiet assumption baked into how most agents handle memory: that more context is safer than less. If the model might need something, put it in the window. The conversation grows, every prior turn rides along on every new request, and we trust the model to find the part that matters. That assumption breaks twice. It breaks on cost , because an agent loop re-sends its whole window on every step — a hundred stale turns aren't paid for once, they're paid for on turn 101, 102, and every step after. And it breaks on quality , because models don't read a long window evenly. Relevant facts buried in the middle get underweighted; irrelevant bulk competes for attention with the thing that actually answers the question. Past a point, a bigger context produces a worse answer, not just a costlier one. So the interesting question isn't "how do we fit more in?" It's "how do we keep the window small and dense without losing the one old turn that matters?" This post is the design we built around that question — for the specific case of long conversation history — plus the benchmark we used to keep ourselves honest. 01 · The mechanism: a hot index over a cold store The design borrows directly from how computers have always managed memory that doesn't fit: a small fast tier that's always present, a large slow tier that holds the bulk, and a rule for moving things between them. Virtual memory pages between RAM and disk. We page between the context window and an external store — for attention instead of address space. Concretely, there are two tiers. The cold store holds every turn at full fidelity, keyed by id — nothing is thrown away. The hot index holds one compact entry per turn: a short summary, a little metadata (entities, whether the turn recorded a dec

If you want to reproduce my current local Hermes Agent + Qwen3.6-27B setup, this is the shape I would start from. Target One local coding agent. One 24GB GPU. Long context. Tools enabled. Thinking enabled. No child agents fighting the main request. The goal is not peak tok/s on a short prompt. The goal is: can the same agent session keep working after hours of tool calls without losing prefix locality, timing out during prefill, or getting wrecked by auxiliary requests? Model This setup is intentionally text-only. I am not serving the multimodal GGUF variant here. The working configuration uses groxaxo/Qwen3.6-27B-GPTQ-Pro-4bit through vLLM with --language-model-only . That choice matters. On a 24GB RTX 3090, the text-only GPTQ-Marlin path gave the best balance I found between long context, prefix caching, stable agent behavior and usable decode speed. Vision should be handled by a separate service/model if needed. vLLM The useful shape: CUDA_VISIBLE_DEVICES = 0 vllm serve groxaxo/Qwen3.6-27B-GPTQ-Pro-4Bit \ --served-model-name qwen3.6-27b-gptq-pro-4bit \ --dtype float16 \ --quantization gptq_marlin \ --tensor-parallel-size 1 \ --max-model-len 131072 \ --max-num-seqs 1 \ --kv-cache-dtype fp8_e5m2 \ --enable-prefix-caching \ --reasoning-parser qwen3 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --gpu-memory-utilization 0.95 \ --max-cudagraph-capture-size 32 \ --language-model-only I used a recent vLLM nightly, not an old stable image ( 0.20.1rc1.dev16+g7a1eb8ac2 ). The two flags people will want to argue about: --max-num-seqs 1 --max-model-len 131072 I use max_num_seqs=1 deliberately. With an agent, parallelism is not free. Title generation, context compression, retries, browser checks, tool calls and side jobs can all steal KV/cache locality from the main request. On one 24GB GPU I prefer one useful request over two requests sabotaging each other. 131k context is tight, but workable here. If your service OOMs, reduce context before adding MTP or enf

Your retriever returned the right documents. The similarity scores look fine. The answer is still wrong. If you've shipped RAG, you've seen this — and it's the failure that survives every retrieval upgrade. What everyone tries Reranker. Higher top-k. Hybrid search. A better embedding model. All of these chase the same goal: documents more similar to the query. They help when the right document wasn't being retrieved. They do nothing when the right document was retrieved and the answer is still wrong. Why it doesn't work Similarity answers "is this chunk about the same topic?" It does not answer "does this chunk contain the facts needed to support the answer?" Those come apart constantly. A chunk can be highly similar — same vocabulary, same subject — and contain nothing that actually grounds the answer. Hand the model a pile of on-topic text and it will produce a fluent, plausible, even cited-looking answer. The grounding is cosmetic: the text was nearby, not load-bearing. High similarity with a wrong answer isn't a contradiction. You asked retrieval to find related text. It did. Nobody asked whether the text was enough. The one shift Stop treating retrieval output as evidence. Treat it as candidate material that has to pass an explicit evidence check before it can support an answer. Put a step between retrieval and generation: does the retrieved set actually contain the facts this answer requires? If not, abstain. When the documents don't contain the facts, the system should return nothing rather than a confident guess. Relevant context in, only sufficient evidence allowed through. That's the line between a RAG demo and a RAG system you can trust in production. I write about the three boundaries where production RAG dies — query, evidence, output — from the angle of shipping under security and model constraints. Read the full version on my blog , where this connects to the practical RAG Failure Diagnosis Kit for teams debugging production RAG.

LLM costs scale linearly with usage. A system processing 10,000 requests a day at $0.01 per request costs $100 daily — $365 a year. At enterprise scale, that's over $10,000. Cost optimization isn't about cutting corners. It's about spending tokens where they matter. Every token you waste is a token you could have spent on a better answer. Token budgeting The simplest way to control costs is to set limits. Per session, per task, or per day. Strategy 1: Per-Session Budgets Per-session budgets are straightforward: class SessionBudget : def __init__ ( self , budget_tokens : int = 10000 ): self . budget = budget_tokens self . used = 0 def allocate ( self , tokens : int ) -> bool : if self . used + tokens <= self . budget : self . used += tokens return True return False def remaining ( self ) -> int : return self . budget - self . used Strategy 2: Per-Task Budgets Per-task budgets are more useful. Different tasks need different amounts of context: task_budgets : classify : max_tokens : 100 model : qwen2.5-1.5b summarize : max_tokens : 500 model : qwen2.5-7b code_review : max_tokens : 2000 model : qwen2.5-coder-7b reason : max_tokens : 4000 model : qwen2.5-32b Strategy 3: Adaptive Budgets Adaptive budgets adjust based on what actually happens. If classification tasks consistently use 80 tokens, stop allocating 100: class AdaptiveBudget : def __init__ ( self ): self . task_history = {} def allocate ( self , task_type : str ) -> int : if task_type in self . task_history : return int ( self . task_history [ task_type ] * 1.5 ) return 1000 def record ( self , task_type : str , tokens_used : int ): if task_type not in self . task_history : self . task_history [ task_type ] = tokens_used else : self . task_history [ task_type ] = ( 0.9 * self . task_history [ task_type ] + 0.1 * tokens_used ) The exponential moving average (0.9 weight) means recent usage matters more than history. Adjust the weight based on how volatile your workloads are. API vs local inference Local inference

LLMs are unpredictable. They hallucinate, leak data, generate harmful content, or refuse legitimate requests. Guardrails constrain model behavior without sacrificing capability. The key is knowing which guardrails matter and which are just noise. Guardrails aren't about controlling the model. They're about controlling the risk. Input validation The most important guardrail. Bad input gets bad output, and bad input can also prompt-inject your system. Strategy 1: Prompt Sanitization Sanitize dangerous patterns early: import re class PromptSanitizer : def __init__ ( self ): self . dangerous_patterns = [ r " ignore\s+previous\s+instructions " , r " system\s+prompt " , r " you\s+are\s+now\s+free " , r " break\s+out\s+of " , ] def sanitize ( self , prompt : str ) -> str : for pattern in self . dangerous_patterns : prompt = re . sub ( pattern , " [REDACTED] " , prompt , flags = re . IGNORECASE ) return prompt This isn't bulletproof. Adversarial inputs are creative. But it catches the obvious ones, and the obvious ones are the most common. Strategy 2: Input Length Limits Length limits prevent token waste and timeouts: class InputValidator : def __init__ ( self , max_length : int = 10000 ): self . max_length = max_length def validate ( self , prompt : str ) -> tuple [ bool , str ]: if len ( prompt ) > self . max_length : return False , f " Input too long: { len ( prompt ) } > { self . max_length } " return True , " OK " Strategy 3: Content Filtering Content filtering blocks policy violations. The patterns here depend on your domain: class ContentFilter : def __init__ ( self ): self . blocked_topics = [ " violence " , " hate speech " , " self-harm " , " sexual content " , " illegal activities " , ] def filter ( self , prompt : str ) -> tuple [ bool , str ]: prompt_lower = prompt . lower () for topic in self . blocked_topics : if topic in prompt_lower : return False , f " Blocked: { topic } " return True , " OK " Simple string matching is fast but imprecise. For production, us