标签：#LLM

Originally published at tengli.dev Your MCP server can be 100% spec-compliant and still be unusable by an agent. The Model Context Protocol spec tells you how to transport tools: JSON-RPC framing, capability negotiation, schema shapes. It says nothing about whether a model can actually use what you serve — whether it picks the right tool out of your catalog, fills the arguments correctly, or burns 8k tokens parsing your schemas on every single request. I integrate first- and third-party MCP connectors into a production AI agent for a living, and I kept seeing the same failure: servers that pass every compliance check, yet the model calls the wrong tool, hallucinates arguments, or ignores the tool entirely. The problems were never in the protocol layer. They were in the parts no one lints: descriptions, naming, schema design. So I wrote mcpgrade — a Lighthouse-style scorecard for MCP servers. One command, no API key, report in seconds: npx mcpgrade --stdio "npx -y your-mcp-server" Then I pointed it at 36 popular servers. It did not go great. The results Full sortable table: https://tengli.dev/mcp-leaderboard.html . The short version (static analysis, point-in-time snapshot; servers marked (archived) are unmaintained reference implementations, included because they're still widely installed and copied): Top of the class (A): brave-search (archived) , exa, google-maps (archived) , slack (archived) , perplexity-ask, @shopify/dev-mcp, @apify/actors-mcp-server, airbnb, figma-developer-mcp, tavily, gitlab (archived) , elastic, shrimp-task-manager, and more — 15 of 36. Bottom of the class (D/F), 11 of 36 — and it's not hobby projects: MongoDB's official server (66, with 66 errors), Notion's official server (62), Airtable (69, 66 errors), todoist-mcp-server (67, 110 errors ), GitHub's archived reference server (67, 44 errors), and firecrawl-mcp at the very bottom (57, 134 errors ). Two more servers (Stripe, Supabase) couldn't be scanned with dummy credentials and were exclud

Every team shipping an agent has the same meeting. Someone asks "is it ready?" and the room splits. One person saw a great demo. Another watched it invent a refund policy an hour ago. The argument runs in circles because nobody agreed what "ready" means, so the loudest opinion wins and the agent ships on a vibe. Making an AI agent production-ready is not a moment of confidence. It is a bar you write down before you build, then measure against. This post is about that bar: why agents need a different one than the services you already ship, and how to define it so "is it ready?" becomes a number instead of an argument. Why "production-ready" breaks for agents For a normal service, "production-ready" is settled. Correct output for valid input, handles errors, meets a latency target, has tests and a rollback. You know the shape of done. An agent breaks three of those assumptions at once: It is non-deterministic. The same input can produce different output, so "correct" becomes "acceptably right, often enough." Its failure surface is open-ended. A function fails in ways you enumerated; an agent fails in ways you never imagined, because it composes language, tools, and judgment on the fly. Its worst case is not a 500 error. It is a confident wrong answer that looks right, which is far more expensive than a crash, because a crash at least tells you it failed. So the honest question is not "is the agent correct." It is "is the agent acceptably wrong, safely, within budget, and repeatably enough to trust." That question has four parts, and each is a line on your bar. The four lines of the bar Write these down before you build. If you cannot fill them in, you do not have a spec, you have a wish. Task success. On a fixed set of real tasks , not the happy-path demo, what fraction must the agent complete correctly? Pick the number. 85 percent means one in seven users gets a wrong answer. Acceptable for this job, or fireable? Decide on purpose. Failure acceptability. Not all wron

Who the F*** Am I? Hi, I'm Daniel Flores. A software developer with, I believe, seven years of professional experience. It's been a wild ride — at least the past three years. I was one of the first to actually try GitHub Copilot during its preview, probably around 2021 or 2022. I was completely amazed by it, but it was definitely very, very rough around the edges. Nobody knows me, though. I never intended to become a public figure or one of those guys who "knows where AI should go." That's not me, and that's okay. I've been watching this new paradigm evolve — and devolve — from the sidelines. What I Think About LLMs and "AI" I've been watching videos about this topic for years. Evolution simulators, learning algorithms, a model trained to play hide and seek — I was completely baffled when I realized that video came from OpenAI itself. I'm not an expert in how to build models or LLMs. I have no PhD. I've just been doing my own thing for years, watching which tools get adopted and which ones suck. And I have to say this: the industry doesn't know what the hell is happening. Neither do I. Nobody knows, and that's what I hate the most. LLMs are extremely useful. They can save you a lot of hours of work. But that's only half the story. So What's the Issue? Almost everyone — paid AI shills, mostly — is telling you: "YOU'RE GETTING BEHIND IF YOU DON'T USE THESE TOOLS!!" They're trying to push the entire industry into a fear-of-missing-out state. Let me share my personal experience about this completely inconsequential fear. If you're already experienced enough — if you already know what an LLM is and can prompt it to do or refactor something — you're not missing out. That's all there is to it. I'm going to explain what I mean. The exact same issues I had with the old GitHub Copilot — I don't even know what model it ran, probably a customized GPT — are still true today with the latest frontier models. They all hallucinate. They all seem to kind of understand what they're do

Most agent tutorials stop at tools: give the model a function schema, hope it calls the right one. That works for get_time and hash_string . It falls apart when the model skips a knowledge search and opens a ticket, or when eighty APIs all land in one context window. Solon AI keeps tools as the execution unit, then adds Talent as the product unit: tools plus SOP plus activation rules. Think of it this way: Tool ≈ a function Talent ≈ a class that owns those functions, their playbook, and when they appear This post is a practical map of when to stay on tools, when to wrap them in a talent, and how registration actually works in Solon v4.0.3. The product failure behind “just add more tools” Bare tools only answer two questions for the model: what can I call? what args does it need? They do not answer: should this capability even be visible right now? what order must I follow before a dangerous call? which tools belong to the same business domain? That gap shows up as: premature side effects (ticket created before diagnosis) context blow-up (full tool tables on every turn) weak SOP compliance (model freestyles across domains) Talent is Solon’s answer: a reusable package of awareness + instruction + tools , with automatic coloring so tools keep their domain identity. Tool vs Talent in one table From the official comparison: Dimension Tool ( FunctionTool ) Talent ( Talent ) Unit Single function / method Instruction + tool set + state Abstraction Physical: how Logical: when and under which SOP Context awareness Passive isSupported(Prompt) can activate or hide Injected content Tool schema (JSON) System prompt fragment + tool list Constraint strength Weak — model freestyles from description Strong — SOP via getInstruction Registration defaultToolAdd / toolAdd defaultTalentAdd / talentAdd They are not rivals. A talent contains tools. Registering a talent also registers its tools; you do not need a second defaultToolAdd for the same set. Lifecycle: what actually runs at reques

Open-source LLMs stopped being the budget option in 2026. Kimi K3 sits level with Claude Opus 4.8 on the Artificial Analysis Intelligence Index (its hosted API is live; the weights themselves are expected by July 27), GLM-5.2 held the top open-model spot before it, and the field behind them is deep enough that the hard part is choosing. This ranking covers the nine best open-weight models right now — on license, context window, hardware reality, and the per-token price you actually pay. Every one of them is available through LLM Gateway with one key, at each provider's published rate, so you can A/B any two of them by changing one word in a request. 1. Kimi K3 — the open frontier Moonshot AI · 2.8T params · 1M context · $3.00 / $15.00 per M The largest open-weight model ever announced — with one caveat: the weights are not downloadable yet. Moonshot expects to release them by July 27, 2026, and the license is still unannounced; the hosted API has been live since July 16. Ranks 4th of 189 models on the Artificial Analysis Intelligence Index — tied with Claude Opus 4.8 and GPT-5.5 — and took first place in Arena's blind Frontend Code testing. Always-on reasoning, vision, tools, and output configurable up to 1M tokens. The open model to beat, priced accordingly. Full breakdown here . Best for: teams that want closed-frontier quality with open-weight freedom. 2. GLM-5.2 — the value flagship Z.ai · 744B params · 1M context · $1.40 / $4.40 per M MIT-licensed, weights on Hugging Face, and the top-ranked open model until K3 arrived. A real 1M-token context, strong agentic-coding results, and built-in web search support — with output at under a third of K3's rate and input at about half. Also the largest model on this list that fits a single 8-GPU node (or one 512 GB Mac Studio) at INT4. Best for: the best capability-per-dollar in the open field. 3. DeepSeek V4 Pro — frontier scale at commodity prices DeepSeek · 1.6T params (49B active) · 1M context · $0.435 / $0.87 per M MI

Kimi K3 took first place in Arena's Frontend Code evaluation the week it launched, and it holds a 1M-token context — but Moonshot doesn't ship a coding agent, and your coding agent doesn't ship Kimi K3. Claude Code is locked to Anthropic's API by default, Cursor to its own backend, Cline to whatever key you hand it. LLM Gateway bridges that gap. It speaks both the Anthropic and OpenAI API formats, so the tools you already use can run Kimi K3 — or any of 200+ models — with a base-URL change. Here is the exact setup for each tool. Kimi K3 in Claude Code Claude Code talks to any endpoint that speaks Anthropic's /v1/messages format, which LLM Gateway does natively. Three environment variables: export ANTHROPIC_BASE_URL = https://api.llmgateway.io export ANTHROPIC_AUTH_TOKEN = $LLM_GATEWAY_API_KEY export ANTHROPIC_MODEL = kimi-k3 claude That's the whole migration. Every request now routes through LLM Gateway to Kimi K3, and every request shows up in your dashboard with its exact cost, token counts, and cache-hit rate. One refinement worth adding: Claude Code uses a second, smaller model for routine background work, and you can point it at something cheap — or free: export ANTHROPIC_SMALL_FAST_MODEL = glm-4.7-flash-free That puts K3 on the hard reasoning and a $0 model on the housekeeping. Kimi K3 in Cursor Cursor routes its chat / plan panel (Cmd/Ctrl + L) through a custom OpenAI-compatible endpoint. Setup: Open Cursor Settings → Models Add your LLM Gateway key under OpenAI API Key Enable Override OpenAI Base URL and set it to https://api.llmgateway.io/v1 Add kimi-k3 as a custom model and select it Be aware of the boundary: Cursor's Composer, inline edit (Cmd/Ctrl + K), and autocomplete are locked to Cursor's own backend and will not route through any external endpoint. Plan and chat with K3's full 1M context in Cursor; if you want K3 driving the actual agent loop, use Claude Code or Cline instead. Kimi K3 in Cline Cline is the straightforward one — it's built to bring y

Kimi K3 is the first open-weight model that makes the Kimi K3 vs Claude Opus 4.8 question worth asking seriously. On the Artificial Analysis Intelligence Index, K3 ranks fourth of 189 models — level with Opus 4.8 and GPT-5.5, behind only Claude Fable 5 and GPT-5.6 Sol. It also costs 40% less per token, and its weights are expected to be released by July 27. The honest answer is not "K3 wins" or "Opus wins" — it depends on what you're optimizing for. Here are the numbers, then the verdict. Benchmarks: Kimi K3 vs Claude Opus 4.8 Benchmark Kimi K3 Claude Opus 4.8 Artificial Analysis Intelligence Index 57 (4th/189) statistical tie GPQA Diamond 93.5% 93.6% Terminal-Bench 2.1 88.3%* 74.6%* SWE-bench Verified not published 88.6% Arena Frontend Code 1st (1,679) ranked below *Reported on different harnesses (Opus 4.8's score uses the Terminus-2 harness), so treat the Terminal-Bench gap as directional, not exact. Three takeaways. On graduate-level reasoning (GPQA Diamond) the models are statistically tied. On agentic terminal work, K3's published number is well ahead, with the harness caveat above. And in Arena's blind Frontend Code testing, developers ranked K3 first outright — ahead of every model, including Anthropic's. Where Opus 4.8 keeps an edge: SWE-bench Verified at 88.6% is a published, battle-tested result K3 has no counterpart for yet, and Anthropic's models remain the default target for agent harnesses — Claude Code, MCP tooling, and most agentic scaffolds are tuned against Claude first. K3 is three days old; its ecosystem is not. Pricing: 40% cheaper across the board Per-token rates through LLM Gateway (each provider's published pricing): Per million tokens Kimi K3 Claude Opus 4.8 Input $3.00 $5.00 Cached input $0.30 $0.50 Output $15.00 $25.00 Both have a 1M-token context window. K3's output limit defaults to 131K tokens but is configurable up to the full 1M in a single response; Opus 4.8 caps output at 128K. Concrete math: a coding-agent workload of 100M input a

Moonshot AI released Kimi K3 on July 16, and the benchmarks put an open-weight model next to the best closed ones for the first time. The catch is access. K3 sits on Moonshot's platform, GLM-5.2 on Z.ai's, DeepSeek V4 Pro on DeepSeek's, MiniMax M3 on MiniMax's — four accounts, four billing relationships, four API dashboards, all before you have written a line of code. LLM Gateway routes every one of them through a single OpenAI-compatible endpoint. One key, one bill, and a switch between Kimi K3 and any of 200+ models is a one-word change to your request. What is Kimi K3? Kimi K3 is Moonshot AI's flagship model for long-horizon coding and agentic work. At 2.8 trillion parameters — a mixture-of-experts design that activates 16 of its 896 experts per token — it is the largest open-weight model announced to date. Moonshot has committed to publishing the full weights by July 27, 2026. The specs that matter in practice: 1M-token context window (1,048,576 tokens), with output configurable up to the same 1M — enough to hold a large repository plus its docs in a single request Always-on reasoning — K3 thinks before every answer; there is no non-thinking mode Vision, tool calls, and JSON output supported out of the box Prompt caching at a 90% discount on repeated input Early results back up the size. K3 ranks fourth of 189 models on the Artificial Analysis Intelligence Index — level with Claude Opus 4.8 and GPT-5.5 — and took first place in Arena's Frontend Code evaluation in blind developer testing. It posted 93.5% on GPQA Diamond and 88.3% on Terminal-Bench 2.1, the strongest open-weight results published on both at release. Kimi K3 pricing Through LLM Gateway you pay Moonshot's published per-token rates: Tokens Price per million Input $3.00 Cached input $0.30 Output $15.00 The cached-input rate is the number to watch. Coding agents re-send the same system prompt, file context, and conversation history on every step, so in a long agent session most of your input tokens are

Building AI Agents That Don't Hallucinate: Structured Workflows, Guardrails, and Per-Step Evaluation How we replaced fragile prompt chains with typed schemas, validation gates, and evaluation at every step — 94% task success vs 60% baseline The Prompt Chain Trap January 2024. We built a "research agent" — 12 prompts chained together: Decompose question → 2. Search planning → 3. Execute searches → 4. Extract facts → 5. Synthesize → 6. Fact-check → 7. Format → ... It worked 60% of the time. The other 40%: Step 3 returned malformed JSON → Step 4 crashed Step 5 hallucinated citations → Step 6 missed it Step 7 output wrong format → Downstream consumer failed No visibility into which step failed Debugging meant reading 12 LLM calls' worth of logs. Adding a step broke three others. The Shift: Agents as Typed Workflows We moved from prompt chains to structured workflows with: Pydantic schemas for every step input/output Guardrails that validate and auto-retry Explicit state machine (not implicit chaining) Evaluation harness per step (not just end-to-end) ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ Decompose │──▶│ Search │──▶│ Extract │──▶│ Synthesize │ │ Question │ │ Planning │ │ Facts │ │ Answer │ │ │ │ │ │ │ │ │ │ In: Query │ │ In: Plan │ │ In: Results │ │ In: Facts │ │ Out: SubQ[] │ │ Out: Steps │ │ Out: Fact[] │ │ Out: Answer │ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │ │ │ │ ▼ ▼ ▼ ▼ [Schema] [Schema] [Schema] [Schema] [Guardrail] [Guardrail] [Guardrail] [Guardrail] [Eval: 0.9] [Eval: 0.85] [Eval: 0.9] [Eval: 0.95] Core Abstractions # agent_eval/schemas.py from pydantic import BaseModel , Field from typing import Literal , Any class DecomposeInput ( BaseModel ): user_query : str context : dict = Field ( default_factory = dict ) class DecomposeOutput ( BaseModel ): sub_questions : list [ str ] = Field ( min_length = 1 , max_length = 5 ) requires_tools : bool reasoning : str class PlanInput ( BaseModel ): sub_questions : list [

You ask an LLM for JSON. You get this: json { "name": "test", "valid": True, "items": [1, 2, markdown Three problems in one response: markdown fences the LLM wasn't supposed to add, True instead of true (Python literal), and the response was cut off mid-array. JSON.parse() throws on all three. A linter tells you what's wrong. But you're left fixing it manually. I kept hitting this wall so often that I built a dedicated repair pipeline: AI JSONMedic . Why existing tools don't cut it JSONLint / JSONFormatter — great for valid-ish JSON with one missing comma. Not built for LLM failure modes: they flag errors but don't repair them. jsonrepair (npm) — solid library, handles many cases. AI JSONMedic actually uses it as a last-resort fallback. But it doesn't tell you what it changed, and doesn't handle all the LLM-specific cases we needed. The 14 failure modes we target Through building this, we catalogued how LLMs specifically break JSON: Markdown fences — ` json wrapping the output Trailing commas — [1, 2, 3,] (the model "runs out" of items but adds one more comma) Python literals — True , False , None instead of true , false , null Single quotes — {'key': 'value'} instead of double quotes Smart quotes — "key" (curly quotes from copy-paste) Unclosed brackets — truncated at max_tokens mid-array or mid-object Unclosed strings — "value without closing Concatenated objects — {"a":1}{"b":2} when streaming produces multiple chunks NDJSON — newline-delimited JSON that needs wrapping Python-style comments — # this is a comment inside JSON JavaScript-style comments — // inline or /* block */ Escaped backslashes — \\n instead of \n Duplicate keys — same key appearing twice (ambiguous — we warn, not silently pick) BOM / encoding issues — UTF-8 BOM at start of response What makes the repair pipeline different Each pass targets one failure mode. The order matters — strip fences first, then normalize quotes, then fix commas, then close truncated structures. Each change is tracked. The

Building AI Agents That Don't Hallucinate: Structured Workflows, Guardrails, and Per-Step Evaluation How we replaced fragile prompt chains with typed schemas, validation gates, and evaluation at every step — 94% task success vs 60% baseline The Prompt Chain Trap January 2024. We built a "research agent" — 12 prompts chained together: Decompose question → 2. Search planning → 3. Execute searches → 4. Extract facts → 5. Synthesize → 6. Fact-check → 7. Format → ... It worked 60% of the time. The other 40%: Step 3 returned malformed JSON → Step 4 crashed Step 5 hallucinated citations → Step 6 missed it Step 7 output wrong format → Downstream consumer failed No visibility into which step failed Debugging meant reading 12 LLM calls' worth of logs. Adding a step broke three others. The Shift: Agents as Typed Workflows We moved from prompt chains to structured workflows with: Pydantic schemas for every step input/output Guardrails that validate and auto-retry Explicit state machine (not implicit chaining) Evaluation harness per step (not just end-to-end) ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ Decompose │──▶│ Search │──▶│ Extract │──▶│ Synthesize │ │ Question │ │ Planning │ │ Facts │ │ Answer │ │ │ │ │ │ │ │ │ │ In: Query │ │ In: Plan │ │ In: Results │ │ In: Facts │ │ Out: SubQ[] │ │ Out: Steps │ │ Out: Fact[] │ │ Out: Answer │ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │ │ │ │ ▼ ▼ ▼ ▼ [Schema] [Schema] [Schema] [Schema] [Guardrail] [Guardrail] [Guardrail] [Guardrail] [Eval: 0.9] [Eval: 0.85] [Eval: 0.9] [Eval: 0.95] Core Abstractions # agent_eval/schemas.py from pydantic import BaseModel , Field from typing import Literal , Any class DecomposeInput ( BaseModel ): user_query : str context : dict = Field ( default_factory = dict ) class DecomposeOutput ( BaseModel ): sub_questions : list [ str ] = Field ( min_length = 1 , max_length = 5 ) requires_tools : bool reasoning : str class PlanInput ( BaseModel ): sub_questions : list [

Optimizing RAG at Scale: Chunking, Retrieval, and the Bayesian Search That Cut Latency 40% How we moved from "semantic search + hope" to a measured, tunable retrieval pipeline with 95% recall@10 The RAG Reality Check Everyone ships RAG the same way: chunk by 512 tokens, embed with text-embedding-3-small , top-k=5, stuff into context. It works for demos. Then you hit production: Legal contracts: 512 tokens splits clauses mid-sentence API docs: 1000-token chunks drown signal in noise Customer tickets: Conversational context needs overlap, not fixed windows Latency: 500ms embedding + 200ms vector search + 300ms LLM = 1s+ per query We rebuilt our retrieval layer from first principles. Here's what actually moves metrics. Chunking: One Size Fits None # rag/chunking.py from abc import ABC , abstractmethod from dataclasses import dataclass @dataclass class Chunk : text : str metadata : dict token_count : int chunk_id : str class ChunkingStrategy ( ABC ): @abstractmethod def chunk ( self , document : str , metadata : dict ) -> list [ Chunk ]: ... class FixedTokenChunker ( ChunkingStrategy ): """ Baseline. Good for homogeneous content. """ def __init__ ( self , chunk_size = 512 , overlap = 50 ): self . chunk_size = chunk_size self . overlap = overlap class RecursiveChunker ( ChunkingStrategy ): """ Respects structure: markdown headers, code blocks, paragraphs. """ def __init__ ( self , separators = [ " \n ## " , " \n ### " , " \n\n " , " \n " , " " ], chunk_size = 512 ): self . separators = separators self . chunk_size = chunk_size class SemanticChunker ( ChunkingStrategy ): """ Uses embedding similarity to find natural boundaries. """ def __init__ ( self , model = " text-embedding-3-small " , threshold = 0.7 ): self . model = model self . threshold = threshold class AgenticChunker ( ChunkingStrategy ): """ LLM decides boundaries. Expensive but highest quality for complex docs. """ def __init__ ( self , model = " gpt-4o-mini " ): self . model = model Our production config by

Agent Determinism Illusions (Part 6) Where this fits: Part 5 closed the experimental arc with an honest answer — no clean fix for the 75% false-negative wall. The Red Line Principle asked the upstream question (when does the loop stop?). This part takes the downstream turn Part 5 already pointed at: stop trying to move the wall; put rules where rules work, LLM only on residual, humans where models diverge. Five insights from overlapping commenters named the pieces (Alexey and Manuel each appear in more than one). Experiment F (38 scenarios) checks whether the resulting pipeline behaves as claimed. Six experiments, 260+ API calls, 15 scripts. Part 5 ended that stretch with: there's no clean solution to LLM output verification. But after those posts went live, commenters saw something I didn't — not gaps in the data, but an architecture I'd failed to draw from my own results. This article collects their five key insights and shows how they reorganize the experiment data into a working pipeline. §§1–4 are paired with experimental or simulation checks from a new prototype (Experiment F, 38 scenarios across two test sets). §5 is a design claim — flagged as such in place. 1. Alexey Spinov & Manuel Bruña: Layer Before You Judge Alexey's comment identified the most fundamental design flaw in my experiments: "G4 ('0 passed, no tests collected') is a fact that can be verified with code in one shot. There is no need to wait for an LLM." Manuel added the constructive direction: "Run deterministic checks first. Then let the LLM handle only the truly ambiguous residual." I went back to my own 8-scenario P1 test set. Four garbage scenarios (G1-G4) and four legitimate ones (L1-L4): ID Output Type Could code catch it? G1 "I am a little duck, quack quack" nonsense ✅ very short + no keywords G2 "。" (a period) pure punctuation ✅ punctuation ratio > 50% G3 "TODO" placeholder ✅ keyword blacklist G4 "0 passed in 0.00s (no tests collected)" zero-test pass ✅ regex 0 passed + no tests All fo

Optimizing RAG at Scale: Chunking, Retrieval, and the Bayesian Search That Cut Latency 40% How we moved from "semantic search + hope" to a measured, tunable retrieval pipeline with 95% recall@10 The RAG Reality Check Everyone ships RAG the same way: chunk by 512 tokens, embed with text-embedding-3-small , top-k=5, stuff into context. It works for demos. Then you hit production: Legal contracts: 512 tokens splits clauses mid-sentence API docs: 1000-token chunks drown signal in noise Customer tickets: Conversational context needs overlap, not fixed windows Latency: 500ms embedding + 200ms vector search + 300ms LLM = 1s+ per query We rebuilt our retrieval layer from first principles. Here's what actually moves metrics. Chunking: One Size Fits None # rag/chunking.py from abc import ABC , abstractmethod from dataclasses import dataclass @dataclass class Chunk : text : str metadata : dict token_count : int chunk_id : str class ChunkingStrategy ( ABC ): @abstractmethod def chunk ( self , document : str , metadata : dict ) -> list [ Chunk ]: ... class FixedTokenChunker ( ChunkingStrategy ): """ Baseline. Good for homogeneous content. """ def __init__ ( self , chunk_size = 512 , overlap = 50 ): self . chunk_size = chunk_size self . overlap = overlap class RecursiveChunker ( ChunkingStrategy ): """ Respects structure: markdown headers, code blocks, paragraphs. """ def __init__ ( self , separators = [ " \n ## " , " \n ### " , " \n\n " , " \n " , " " ], chunk_size = 512 ): self . separators = separators self . chunk_size = chunk_size class SemanticChunker ( ChunkingStrategy ): """ Uses embedding similarity to find natural boundaries. """ def __init__ ( self , model = " text-embedding-3-small " , threshold = 0.7 ): self . model = model self . threshold = threshold class AgenticChunker ( ChunkingStrategy ): """ LLM decides boundaries. Expensive but highest quality for complex docs. """ def __init__ ( self , model = " gpt-4o-mini " ): self . model = model Our production config by

Measured 2026-07-21 on vllm/vllm-tpu:nightly (vLLM 0.23.1rc1.dev1076), a GCE flex-start ct6e-standard-1t (one TPU v6e chip, 32 GB HBM) in europe-west4-a. TL;DR The plain google/gemma-4-E2B-it serves well on one v6e chip; none of its QAT siblings load at all. The 2-billion-parameter "efficient" Gemma 4 sustains 213 tok/s for a single user with a 16 ms first token, scales to ~2,200 output tok/s across concurrent streams, handles OpenAI-style function calling — including parallel calls and refusal to hallucinate calls — without a miss, and answers simple vision questions accurately in ~200 ms. The QAT variants are a different story: the int4 compressed-tensors export hits an unimplemented quantization path, and the bf16 QAT export trips a loader bug — the Gemma 4 implementation demands per-layer norms that E2B's KV-sharing architecture legitimately doesn't have. Filed upstream as tpu-inference #3225 . One capability coupling to know about: with a reasoning parser configured, schema enforcement only engages when thinking is enabled — thinking-off requests sail through unconstrained with a 200 status. Config interaction, not TPU limitation; details below. 1. Getting to a serving endpoint The host is a GCE flex-start VM — capacity granted on request, billed until deleted, hard-stopped at a 4-hour max run, $1.35/chip-hour. A startup script installs Docker, pulls vllm/vllm-tpu:nightly , fetches the Hugging Face token from Secret Manager via the metadata server, and launches vLLM. Boot timeline: VM RUNNING at t+0 (200 GB boot disk — the 10 GB default cannot hold the vLLM image) → Docker installed ~t+1:00 → image pulled ~t+6:00 → weights downloaded, XLA compiled, health green ~t+8:30. Two environment quirks worth knowing: Direct SSH silently times out on some networks even when the VPC allows tcp:22 — the block is upstream of the VPC. IAP tunneling ( gcloud compute ssh --tunnel-through-iap ) rides over HTTPS and works; so does tunneling the API port with gcloud compute start-

Building Production-Grade LLM Evaluation Pipelines: From Vibes to Metrics How we replaced "looks good to me" with automated evaluation catching 92% of hallucinations before deployment The Problem: Why "Vibe Checks" Fail in Production Three months ago, our team shipped a RAG-based customer support assistant. It worked great in testing — we'd ask it questions, read the answers, and say "yeah, that looks right." Then it hit production. A customer asked about their billing cycle. The assistant confidently cited a policy that didn't exist. Another asked about API rate limits and got numbers from a competitor's documentation. By the time we caught it, 500+ users had seen hallucinated responses. The post-mortem was brutal: we had zero automated evaluation . Our test process was literally "ask 5 questions, read answers, thumbs up." What Production Evaluation Actually Needs Academic benchmarks (MMLU, HellaSwag) don't tell you if your system works for your use case. Production evaluation needs: Domain-specific judges — Your criteria, not generic "helpfulness" Speed — Evaluation must run in CI/CD, not overnight Regression detection — Know immediately when a prompt change breaks things CI/CD integration — Block merges that degrade quality Golden dataset management — Versioned, stratified, growing test cases Architecture: The Evaluation Pipeline ┌─────────────┐ ┌──────────────┐ ┌────────────────────┐ ┌──────────────┐ │ Test Cases │────▶│ LLM Under │────▶│ Judge Ensemble │────▶│ Metrics & │ │ (Golden Set)│ │ Test │ │ - Faithfulness │ │ Regression │ └─────────────┘ └──────────────┘ │ - Instruction F. │ │ Detection │ │ - JSON Schema │ └──────┬───────┘ │ - Custom LLM │ ▼ └────────────────────┘ ┌──────────────┐ │ Dashboard/ │ │ PR Comments │ └──────────────┘ Core Abstractions # eval/base.py @dataclass ( frozen = True ) class TestCase : id : str input : dict [ str , Any ] expected : dict [ str , Any ] | None = None tags : list [ str ] = field ( default_factory = list ) # ["edge-case",

Optimizing RAG at Scale: Chunking, Retrieval, and the Bayesian Search That Cut Latency 40% How we moved from "semantic search + hope" to a measured, tunable retrieval pipeline with 95% recall@10 The RAG Reality Check Everyone ships RAG the same way: chunk by 512 tokens, embed with text-embedding-3-small , top-k=5, stuff into context. It works for demos. Then you hit production: Legal contracts: 512 tokens splits clauses mid-sentence API docs: 1000-token chunks drown signal in noise Customer tickets: Conversational context needs overlap, not fixed windows Latency: 500ms embedding + 200ms vector search + 300ms LLM = 1s+ per query We rebuilt our retrieval layer from first principles. Here's what actually moves metrics. Chunking: One Size Fits None # rag/chunking.py from abc import ABC , abstractmethod from dataclasses import dataclass @dataclass class Chunk : text : str metadata : dict token_count : int chunk_id : str class ChunkingStrategy ( ABC ): @abstractmethod def chunk ( self , document : str , metadata : dict ) -> list [ Chunk ]: ... class FixedTokenChunker ( ChunkingStrategy ): """ Baseline. Good for homogeneous content. """ def __init__ ( self , chunk_size = 512 , overlap = 50 ): self . chunk_size = chunk_size self . overlap = overlap class RecursiveChunker ( ChunkingStrategy ): """ Respects structure: markdown headers, code blocks, paragraphs. """ def __init__ ( self , separators = [ " \n ## " , " \n ### " , " \n\n " , " \n " , " " ], chunk_size = 512 ): self . separators = separators self . chunk_size = chunk_size class SemanticChunker ( ChunkingStrategy ): """ Uses embedding similarity to find natural boundaries. """ def __init__ ( self , model = " text-embedding-3-small " , threshold = 0.7 ): self . model = model self . threshold = threshold class AgenticChunker ( ChunkingStrategy ): """ LLM decides boundaries. Expensive but highest quality for complex docs. """ def __init__ ( self , model = " gpt-4o-mini " ): self . model = model Our production config by

Building Production-Grade LLM Evaluation Pipelines: From Vibes to Metrics How we replaced "looks good to me" with automated evaluation catching 92% of hallucinations before deployment The Problem: Why "Vibe Checks" Fail in Production Three months ago, our team shipped a RAG-based customer support assistant. It worked great in testing — we'd ask it questions, read the answers, and say "yeah, that looks right." Then it hit production. A customer asked about their billing cycle. The assistant confidently cited a policy that didn't exist. Another asked about API rate limits and got numbers from a competitor's documentation. By the time we caught it, 500+ users had seen hallucinated responses. The post-mortem was brutal: we had zero automated evaluation . Our test process was literally "ask 5 questions, read answers, thumbs up." What Production Evaluation Actually Needs Academic benchmarks (MMLU, HellaSwag) don't tell you if your system works for your use case. Production evaluation needs: Domain-specific judges — Your criteria, not generic "helpfulness" Speed — Evaluation must run in CI/CD, not overnight Regression detection — Know immediately when a prompt change breaks things CI/CD integration — Block merges that degrade quality Golden dataset management — Versioned, stratified, growing test cases Architecture: The Evaluation Pipeline ┌─────────────┐ ┌──────────────┐ ┌────────────────────┐ ┌──────────────┐ │ Test Cases │────▶│ LLM Under │────▶│ Judge Ensemble │────▶│ Metrics & │ │ (Golden Set)│ │ Test │ │ - Faithfulness │ │ Regression │ └─────────────┘ └──────────────┘ │ - Instruction F. │ │ Detection │ │ - JSON Schema │ └──────┬───────┘ │ - Custom LLM │ ▼ └────────────────────┘ ┌──────────────┐ │ Dashboard/ │ │ PR Comments │ └──────────────┘ Core Abstractions # eval/base.py @dataclass ( frozen = True ) class TestCase : id : str input : dict [ str , Any ] expected : dict [ str , Any ] | None = None tags : list [ str ] = field ( default_factory = list ) # ["edge-case",

Optimizing RAG at Scale: Chunking, Retrieval, and the Bayesian Search That Cut Latency 40% How we moved from "semantic search + hope" to a measured, tunable retrieval pipeline with 95% recall@10 The RAG Reality Check Everyone ships RAG the same way: chunk by 512 tokens, embed with text-embedding-3-small , top-k=5, stuff into context. It works for demos. Then you hit production: Legal contracts: 512 tokens splits clauses mid-sentence API docs: 1000-token chunks drown signal in noise Customer tickets: Conversational context needs overlap, not fixed windows Latency: 500ms embedding + 200ms vector search + 300ms LLM = 1s+ per query We rebuilt our retrieval layer from first principles. Here's what actually moves metrics. Chunking: One Size Fits None # rag/chunking.py from abc import ABC , abstractmethod from dataclasses import dataclass @dataclass class Chunk : text : str metadata : dict token_count : int chunk_id : str class ChunkingStrategy ( ABC ): @abstractmethod def chunk ( self , document : str , metadata : dict ) -> list [ Chunk ]: ... class FixedTokenChunker ( ChunkingStrategy ): """ Baseline. Good for homogeneous content. """ def __init__ ( self , chunk_size = 512 , overlap = 50 ): self . chunk_size = chunk_size self . overlap = overlap class RecursiveChunker ( ChunkingStrategy ): """ Respects structure: markdown headers, code blocks, paragraphs. """ def __init__ ( self , separators = [ " \n ## " , " \n ### " , " \n\n " , " \n " , " " ], chunk_size = 512 ): self . separators = separators self . chunk_size = chunk_size class SemanticChunker ( ChunkingStrategy ): """ Uses embedding similarity to find natural boundaries. """ def __init__ ( self , model = " text-embedding-3-small " , threshold = 0.7 ): self . model = model self . threshold = threshold class AgenticChunker ( ChunkingStrategy ): """ LLM decides boundaries. Expensive but highest quality for complex docs. """ def __init__ ( self , model = " gpt-4o-mini " ): self . model = model Our production config by

Optimizing RAG at Scale: Chunking, Retrieval, and the Bayesian Search That Cut Latency 40% How we moved from "semantic search + hope" to a measured, tunable retrieval pipeline with 95% recall@10 The RAG Reality Check Everyone ships RAG the same way: chunk by 512 tokens, embed with text-embedding-3-small , top-k=5, stuff into context. It works for demos. Then you hit production: Legal contracts: 512 tokens splits clauses mid-sentence API docs: 1000-token chunks drown signal in noise Customer tickets: Conversational context needs overlap, not fixed windows Latency: 500ms embedding + 200ms vector search + 300ms LLM = 1s+ per query We rebuilt our retrieval layer from first principles. Here's what actually moves metrics. Chunking: One Size Fits None # rag/chunking.py from abc import ABC , abstractmethod from dataclasses import dataclass @dataclass class Chunk : text : str metadata : dict token_count : int chunk_id : str class ChunkingStrategy ( ABC ): @abstractmethod def chunk ( self , document : str , metadata : dict ) -> list [ Chunk ]: ... class FixedTokenChunker ( ChunkingStrategy ): """ Baseline. Good for homogeneous content. """ def __init__ ( self , chunk_size = 512 , overlap = 50 ): self . chunk_size = chunk_size self . overlap = overlap class RecursiveChunker ( ChunkingStrategy ): """ Respects structure: markdown headers, code blocks, paragraphs. """ def __init__ ( self , separators = [ " \n ## " , " \n ### " , " \n\n " , " \n " , " " ], chunk_size = 512 ): self . separators = separators self . chunk_size = chunk_size class SemanticChunker ( ChunkingStrategy ): """ Uses embedding similarity to find natural boundaries. """ def __init__ ( self , model = " text-embedding-3-small " , threshold = 0.7 ): self . model = model self . threshold = threshold class AgenticChunker ( ChunkingStrategy ): """ LLM decides boundaries. Expensive but highest quality for complex docs. """ def __init__ ( self , model = " gpt-4o-mini " ): self . model = model Our production config by