标签：#LLM

"Here’s a more natural, flowing version of that section..."

The same bug hit me in three separate sessions before I fixed it properly. Each time, my orchestrator reached a decision point, tried to present its menu, and burned a turn on a validation error instead of a question: InputValidationError: { "code" : "too_big" , "maximum" : 4 , "path" : [ "questions" , 0 , "options" ] } // abridged; the full payload includes the Zod message Claude Code's AskUserQuestion tool caps every question at 4 options. My menu had 5. First strike: the end-of-run menu. Second strike: a blocker-recovery menu. Third strike: the same recovery menu two weeks later, after I thought I'd fixed it. That repetition is the story. Why this one keeps coming back A one-off validation error is not worth a blog post. What makes this one worth writing up is why it recurred: the cause wasn't a typo. It was a template. Suhail , my Claude Code orchestrator, is a set of markdown prompt files, and its menus live in those files as literal option lists: the template says exactly what to present, and the model presents it verbatim. Five options go into one AskUserQuestion call, the schema rejects it, and the round-trip to the model is wasted. In my runs the model then retried with four options and continued, which is why, the first two times, I let the retry count as the fix. The recovery is so cheap that the bug reads as a hiccup, not a defect. But decision menus are the natural accumulation point of any orchestrator. Every new capability wants a slot: continue, commit, skip, retry, abort, show status. The menu only grows. A 5-option template doesn't fail once; it fails on every run that reaches it, one wasted turn each time, until you fix the template. The failure mode worse than the error The wasted turn is the benign version. Suhail's public changelog records the malignant one: the interactive complete-handler menu grew past the cap, and instead of erroring, the presented menu simply lost its last option. The option that got pushed out of reach was Abort . Abort o

Agents that call tools over a network eventually retry a call they shouldn't have. If the tool isn't idempotent, that retry becomes a duplicate charge, a duplicate email, a duplicate order — quietly, with nothing in the logs to flag it. It's a decades-old distributed-systems problem wearing a new agent costume, and as far as I could tell, nobody had shipped a small, pip-installable fix for the agent-shaped version of it. So I built one. It's called latch . Worth saying up front: this is not a novel idea, and I'd rather say so myself than have someone point it out in the comments. The bug, live Here's the whole thing in one file, no library involved: def charge_card ( order_id , amount ): time . sleep ( 0.4 ) # the payment API is a little slow today ledger [ order_id ] += 1 return { " order_id " : order_id , " status " : " charged " } def agent_charge_with_naive_retry ( order_id , amount , max_retries = 2 ): for attempt in range ( max_retries ): thread = threading . Thread ( target = lambda : charge_card ( order_id , amount )) thread . start () thread . join ( timeout = 0.2 ) # the agent's own client-side timeout if thread . is_alive (): continue # "no response, let's retry" return # got a response The payment call takes 0.4s. The agent gives up waiting after 0.2s and retries. The first attempt is still running in the background — it wasn't cancelled, it can't be safely cancelled, Python threads don't work that way. So it finishes on its own time and charges the card. Then the retry charges it again. The agent's own view of the world is "everything's fine, got a response eventually." The ledger says otherwise. This is examples/naive_agent_example.py in the repo, if you want to run it and watch it happen rather than take my word for it. None of this is exotic. It's the same class of problem as "what happens when a payment gateway's webhook fires twice," which every backend engineer who's touched Stripe has dealt with. The fix has a name — idempotency keys — and it's b

Every "run this model locally" guide tells you to grab a Q4 GGUF and move on. That advice is fine right up until you try a long-context run and your machine starts swapping. The weights are the part everyone budgets for Quantization maths is straightforward. A model's weight footprint is roughly params x bits / 8 : Quant Bits/param 12B model Quality note Q8_0 ~8.5 ~12.8 GB Near-lossless, rarely worth it Q6_K ~6.6 ~9.9 GB Very close to Q8 Q4_K_M ~4.8 ~7.2 GB The usual sweet spot Q3_K_M ~3.9 ~5.9 GB Noticeable degradation Below Q4 the loss stops being subtle. Instruction-following degrades before raw perplexity does, which is why benchmark numbers can look fine while the model quietly stops respecting your system prompt. The KV cache is the part that bites Here is what the guides skip. The KV cache scales with context length , and it is not quantized by default: kv_bytes ~= 2 (K and V) x layers x kv_heads x head_dim x seq_len x dtype_bytes The practical consequence: a model that loads in 7 GB can need well over twice that at long context. Grouped-query attention helps a lot — kv_heads is much smaller than attention heads — but the term still grows linearly with sequence length while your weights stay fixed. Two knobs matter more than picking a fancier quant: --ctx-size : do not allocate 128K if your prompts are 8K. You are reserving memory you will never touch. KV cache quantization ( q8_0 for K/V): roughly halves cache memory for a quality hit most workloads never notice. Underused. A decision order that works Start at Q4_K_M Set context to what you actually use, not the model maximum If you are still tight, quantize the KV cache before dropping to Q3 Only move up to Q6/Q8 if you have headroom left over That ordering matters: dropping to Q3 to buy context is the most common mistake, and it trades a permanent quality loss for memory you could have gotten from the cache instead. Per-quantization benchmarks and deployment notes for the Gemma 4 family are collected at ge

A few months ago, if you had asked me whether Structured Outputs solved hallucinations, I probably would have said yes. Today, I wouldn't. Not because Structured Outputs don't work-they absolutely do. But because they solve a different problem . After building ShapeCraft, an open-source structured output library supporting OpenAI, Groq, Ollama (GBNF), and Anthropic, I realized something that fundamentally changed how I think about production AI. Structure and correctness are two completely different guarantees. The misconception Let's say you're extracting data from an invoice. The invoice says: Invoice Total: $900 Your model returns: { "invoiceTotal" : 1200 } Let's validate it. Validation Result Valid JSON ✅ Matches JSON Schema ✅ Required field present ✅ Correct data type ✅ Correct answer ❌ The response is structurally perfect. The extracted value is still wrong. JSON Schema didn't fail. It did exactly what it was designed to do. What JSON Schema actually guarantees JSON Schema is responsible for ensuring your application receives data in a predictable format. It guarantees things like: Valid JSON Required fields Correct data types Nested object structure Array validation Enum validation That makes applications dramatically easier to build. But it does not guarantee: The extracted value is correct The answer exists in the source document The model didn't hallucinate Business rules were followed Those are different problems. Structure ≠ Truth I've started thinking about AI validation as two independent layers. Layer 1 - Structural Validation Can my application safely consume this response? Examples: Is it valid JSON? Does it match my schema? Are required fields present? JSON Schema solves this extremely well. Layer 2 - Semantic Validation Can I trust the information? Questions become: Did the model extract the correct value? Can I trace it back to the source? Is there supporting evidence? Would I make a business decision based on this response? This layer is much ha

There's a whole ecosystem of "agent skills" now — reusable instruction files you drop into Claude Code (or Cursor, or Copilot) to make the model write cleaner code, debug more carefully, use fewer tokens, and so on. Some of these repos have tens of thousands of GitHub stars. Almost none of them ship a single number telling you whether the skill actually does anything. That bothered me, because "adding a plausible-sounding instruction" and "adding an instruction that works" look identical until you measure them. So I built a benchmark with one rule, committed before I ran anything: No skill gets merged unless it beats both a no-instruction baseline AND a placebo prompt on its pre-registered target metric, measured on hidden hold-out tests, with accuracy not allowed to drop. Skills that fail are published anyway, with their numbers. The placebo arm is the part almost nobody runs, and it turned out to be the most important one. Why a placebo Most "battle-tested" skill collections that measure anything at all compare skill-on vs skill-off. The problem: that comparison can't separate "this skill works" from "adding any confident-sounding text changes the model's behavior." LLMs are suggestible. If you want to claim your skill did something, you have to show it beats a same-length instruction that contains no actual mechanism — just vibes. So every result here is a three-way comparison — off / placebo / on — run K=5–8 times per task per arm, in isolated git workspaces, graded by hold-out acceptance tests the agent never sees, with every raw run log committed to the repo and the README regenerated from those logs in CI. 516 runs total, all on claude-opus-4-8 . Finding 1: the placebo often made code bigger My anti-over-engineering skill ( underkill , ~20 lines) cut source LOC by -23.8% vs baseline at identical accuracy (60/60 hold-out passes). Good. But the interesting column is the placebo: a same-length "write clean, minimal, professional code" instruction didn't reduce c

A nova fase dos agentes de IA: menos chat, mais operação. A OpenAI apresentou o Presence, uma plataforma para empresas implantarem agentes de voz e chat em atendimento ao cliente e fluxos internos. Para quem desenvolve sistemas, o ponto não é apenas colocar mais um chatbot em produção. A mudança relevante é tratar a IA como uma camada operacional: ela precisa participar de processos reais, com objetivos claros e resultados verificáveis. Um agente confiável precisa de mais do que boas respostas. Precisa receber o contexto certo, operar dentro de permissões bem definidas, deixar rastros auditáveis e saber quando transferir o caso para uma pessoa. Isso altera a decisão técnica. Escolher um modelo continua importante, mas arquitetura, integrações, observabilidade, avaliação contínua e governança passam a fazer parte do mesmo problema. A pergunta mais útil deixa de ser “onde podemos colocar IA?” e passa a ser: qual processo já está pronto para ser redesenhado com um agente de confiança? Fonte oficial: https://openai.com/index/introducing-openai-presence

This essay was written with Barath Raghavan, and originally appeared in The Guardian . Major benchmarks measure what AI can do. None measure whether it does what you mean: the distance between what you ask an AI to do and the unspoken assumptions about how you want the AI to do it. We propose a new metric: the Genie coefficient. There’s often a gap between one person’s request and another’s understanding. Most of the time, we bridge it using general knowledge. For example, if you ask a friend to get you coffee, they’ll pour a cup from the pot or buy one from a coffee shop. They won’t bring you a bag of raw beans or snatch a cup from a stranger and hand it to you. You never specified any of this. You never had to...

Hetzner is experimenting with LLM inference. That is not a sentence I expected to write, but I think it is pretty interesting :) Before anyone moves their production AI workloads to Hetzner: this is very much an experiment . There is no billing, no SLA, no production guarantee, and currently only one model. Hetzner says it wants to learn whether people actually want this, how the system scales, which features matter, and what kind of load it can handle. So this is not a finished product launch. It is Hetzner putting something early in front of users and seeing what happens. I really like that approach. What Is Hetzner Inference? Hetzner Inference is an OpenAI-compatible API running on Hetzner's own infrastructure. You create an API token in the Experiments dashboard, point an OpenAI client at Hetzner's base URL, and use it like most other inference APIs. Right now, the only available model is Qwen/Qwen3.6-35B-A3B-FP8 . It is a 35-billion-parameter Mixture-of-Experts model with 3 billion active parameters. It accepts text and images, has a 262K context window, and uses FP8-quantized weights. That is a perfectly reasonable model for an experiment. It is small enough to serve without a ridiculous GPU cluster, but still useful enough to test the API with real workloads. Hetzner also published a short tutorial for connecting OpenCode to the API , if you want to try it without writing any code. I Tried It Because the API is OpenAI-compatible, there is almost nothing special about the integration: pip install openai from openai import OpenAI client = OpenAI ( base_url = " https://inference.hetzner.com/api/v1 " , api_key = " YOUR_TOKEN " , ) response = client . chat . completions . create ( model = " Qwen/Qwen3.6-35B-A3B-FP8 " , messages = [ { " role " : " user " , " content " : " Explain why the sky is blue in one sentence. " } ], extra_body = { " chat_template_kwargs " : { " enable_thinking " : False , } }, ) print ( response . choices [ 0 ]. message . content ) The enabl

A validation failure is not an exception to hide. It is a record your system does not yet know how to trust. That distinction matters in LLM extraction pipelines. A malformed invoice, an unexpected OCR layout, a model response that violates the schema, and a semantically impossible value may all reach the same line of validation code. If the only outcomes are “retry” or “drop,” the pipeline will either waste money repeating the same failure or silently lose work. The production answer is a dead-letter path: a durable place for failed records to wait with enough evidence to explain, triage, and safely replay them. The queue itself is the easy part. The hard part is designing the failure contract around it. Validation is where routing begins Constrained decoding and post-hoc validation solve different problems . Even with both, some records should fail. Real documents are messy, schemas change, OCR corrupts values, and models sometimes return plausible nonsense. A robust validation boundary should produce more than true or false . It should emit a reason the rest of the pipeline can act on: which schema and model versions were used which fields failed and why whether the payload was malformed, incomplete, or semantically invalid the confidence signal attached to the extraction whether a retry is likely to change the result That result becomes a routing decision. High-confidence, valid records can flow forward. Recoverable transport failures can use a bounded retry. Ambiguous or invalid records belong in review or a dead-letter queue. Confidence-based routing is useful precisely because “trust everything” and “review everything” are both bad operating models. A dead-letter record needs evidence, not just payload Putting the original input on another queue is not enough. Without context, the team investigating the failure has to reconstruct the run from scattered logs—if those logs still exist. I would store a dead-letter envelope containing: a stable record ID and idem

Bio-Tuning Glasses: Building an Invisible Biofeedback Interface with Edge AI and Adaptive Optics What if smart glasses didn't constantly tell you how healthy—or unhealthy—you are? No step counts. No stress notifications. No endless dashboards. No digital reminders telling you to "sit straight" or "go to sleep." Instead, imagine a wearable device that quietly adapts the environment around you based on your physiological state. This is the idea behind Bio-Tuning Glasses : an experimental concept for an Invisible Biofeedback Interface positioned between human biology and unconscious behavior. The goal is simple: Don't make the user adapt to the technology. Make the environment adapt to the user. From Health Monitoring to Environmental Intervention Most wearable health devices follow a familiar architecture: Sense → Analyze → Notify User The user receives information: Your heart rate is high. You are stressed. You haven't moved enough. Your sleep quality is poor. Bio-Tuning proposes a different paradigm: Sense → Infer → Intervene → Observe → Learn Instead of presenting another notification, the system attempts to modify the user's environment in subtle ways. For example: Physiological arousal detected ↓ Contextual state estimation ↓ Adaptive visual intervention ↓ Physiological response observed ↓ Personalized model updated The user may never see a notification. The intervention simply happens in the background. 1. Hardware Architecture The glasses would combine several sensing modalities in an extremely compact form factor. Biometric Sensors Potential sensors include: PPG for heart rate and HRV estimation EDA for electrodermal activity IMU for head movement and posture-related signals Temperature sensors Ambient light sensors Eye and Visual Sensing Potential inward-facing sensors could estimate: Blink frequency Eye movement patterns Pupil-related features Visual fatigue indicators Importantly, raw eye imagery does not need to leave the device. Instead: Raw Sensor Data ↓

If you build anything on top of hosted AI APIs, latency isn't a detail you get to ignore — it's a feature. A sluggish time-to-first-token is the difference between an assistant that feels alive and one that feels broken. Yet when I went looking for an honest answer to a simple question — how fast is provider X from where my users actually are? — I couldn't find one. The numbers people quote tend to come from a single machine in a single region (usually somewhere in the US), from vendor-reported status pages, or from a benchmark that got run once and never refreshed. There was no independent, measured , regional view. So I built one: LLM Latency Tracker , a provider-neutral tracker of latency and uptime for AI inference APIs. How it works The core idea is boring on purpose: actually measure, don't scrape. A small Python prober — standard library only, no API key needed for the edge probes — opens real connections to each provider's endpoint and times every phase of the handshake: DNS resolution → TCP connect → TLS negotiation → time-to-first-byte (TTFB) . That's the edge view: how long the network path itself takes before a single byte comes back. Separately, where possible, it measures inference time-to-first-token (TTFT) — the thing your users actually feel, i.e. how long after you hit "send" the model starts streaming. Those two numbers answer different questions, and keeping them apart matters. Edge latency is about the network and the front door; TTFT is about the model and the queue behind it. The probes run from four regions — Europe (Germany), US Central, Asia (Tokyo), and South America (São Paulo) — because "fast" is meaningless without "from where." Results land in a SQLite time-series; a static-site generator turns that into the pages you see, hosted on Cloudflare Pages, and the whole thing is self-updating on a schedule. There's no always-on backend to rot or page me at 3am. It currently covers ~45 providers — the usual Western labs (OpenAI, Anthropic, Go

Divergence escalates the wrong population: unanimous misses auto-pass Agent Determinism Illusions (Part 7) Where this fits: This part does not continue Part 13's probe-vs-prose thread. It returns to Part 6 's L2→L3 escalation rule — Dipankar's move of treating vote disagreement as the human-review signal. Alexey Spinov's follow-up comment says that signal points at the wrong population. Two experiments check whether he is right, and what to put in the tripwire instead. Part 6 drew this control flow: L2 multi-perspective votes │ unanimous ──────────► AUTO-PASS / AUTO-REJECT │ divergence (e.g. 2–1) ► L3 human The caveat was already in the text: divergence measures ambiguity; it does not fix unanimous systematic bias. Alexey's point is sharper — and it is about routing , not about another caveat paragraph. 1. Alexey's population mismatch On the Part 6 thread, Alexey Spinov wrote (paraphrased tightly): The dangerous failures are high-confidence and directional — systematic. Systematic bias is shared across prompts, not idiosyncratic (your own P3: majority voting doesn't fix it). So the three perspectives will tend to agree on exactly those cases. Divergence-to-human then routes you the safely-ambiguous ones and auto-passes the confidently-wrong ones. The escalation signal is pointing at the wrong population. He proposed two cheap replacements: T1 — deterministic tripwire on known-reversal classes (escalate regardless of agreement). T2 — treat unanimous + high-confidence on a historically reversal-prone class as escalate — the inverse of “high confidence, auto-pass.” That is the claim under test. Not “divergence is useless,” but “divergence alone is the wrong primary tripwire for the failure mode you already measured.” 2. Experiment A — offline proxy on DF v2 (no new API) Part 6's Mike Update already showed: of 96 DF v2 MISS runs, 95.8% sat at self-reported confidence ≥ 0.9 (avg 0.969). That mass is concentrated — Part 6 also reported ~80% of MISS runs from qwen3:0.5b —

TL;DR : most production AI tasks are not LLM tasks. To triage my email, I replaced a 7-billion-parameter model with a tiny classifier in Go. The rule fits in one sentence. Rules first, a small model next, the LLM only as a last resort. The result: no GPU, sub-millisecond inference, and a cloud call that became rare. Here is how, with the real numbers. This article is for developers who put an LLM in production and pay the bill. Not a demo. Most AI tasks are not LLM tasks In 2026, the default reflex is to wire a big model into everything. A question comes in, you call the LLM. But many tasks do not need it. Filing an email under "work" or "newsletter" is classification. A problem solved for twenty years, long before LLMs. To classify is to pick a label from a short, stable list. To generate text is something else. The first job needs a small model. The second earns a big one. The rule I defend fits in one sentence. Put the LLM last. The setup I built an agent that triages my inbox. It is a daemon. It reads new messages and files each one into a category: work, notification, newsletter, promo, and a few more. Nothing secret, just my real mailbox, with years of mail. The first version handed every email to a local LLM. A 7-billion-parameter model, Qwen 2.5 7B, served by Ollama on a GPU. Ollama is a tool that runs an LLM on your own machine. It worked. But the price was heavy. A GPU on all the time. One more container to watch. And an absurd slowness for the question asked. One day I asked myself: does deciding "is this a newsletter?" really need 7 billion parameters? No. The answer sits in two or three words from the sender and the subject. So I rethought the whole thing. Three layers, from cheapest to most expensive Every email goes through three layers, in order. It stops at the first one that can answer. Deterministic rules. Instant, exact, no cost. A small model. Sub-millisecond, on CPU. The LLM. Only if the small model is unsure. The routing code fits in a few lin

AutoGen's hidden token tax: why a 3-agent chat costs 15× what you expect Cost-audit series, episode 2. This series began with an AI agent that burned 136M tokens overnight → . AutoGen is Microsoft's multi-agent framework. It's genuinely good at orchestrating agents that hand off work to each other. But its default memory model has a cost shape that surprises almost every team that hits it in production. This audit shows you exactly where the tokens go, with line numbers. The setup: a 3-agent RoundRobin chat The canonical AutoGen pattern is a RoundRobinGroupChat with N agents taking turns on a task. Here's the minimal version from the docs: from autogen_agentchat.agents import AssistantAgent from autogen_agentchat.teams import RoundRobinGroupChat from autogen_agentchat.conditions import MaxMessageTermination planner = AssistantAgent ( " planner " , model_client = client , system_message = " You plan. " ) coder = AssistantAgent ( " coder " , model_client = client , system_message = " You code. " ) reviewer = AssistantAgent ( " reviewer " , model_client = client , system_message = " You review. " ) team = RoundRobinGroupChat ( [ planner , coder , reviewer ], termination_condition = MaxMessageTermination ( max_messages = 10 ), ) await team . run ( task = " Build a web scraper for Hacker News. " ) Three agents, 10 turns total (~3–4 turns each). Seems cheap. It isn't. The default context: unbounded, per-agent Every AssistantAgent gets its own UnboundedChatCompletionContext by default: # autogen-agentchat/src/autogen_agentchat/agents/_assistant_agent.py, __init__ (L708) if model_context is not None : self . _model_context = model_context else : self . _model_context = UnboundedChatCompletionContext () source UnboundedChatCompletionContext.get_messages() returns self._messages — the full list, no cap, no truncation: # autogen-core/.../model_context/_unbounded_chat_completion_context.py (a ~20-line file) async def get_messages ( self ) -> List [ LLMMessage ]: """ Get at most

OpenAI, Anthropic, and Google expose different APIs, message formats, tool-calling conventions, error types, and response structures. That difference is manageable while an application uses only one provider. It becomes more expensive when the application needs retries, circuit breakers, fallback routes, observability, and recovery across several providers. Without a shared abstraction, resilience logic tends to be implemented repeatedly: OpenAI integration ├── request conversion ├── retry logic ├── error classification ├── circuit breaker └── tool-call recovery Anthropic integration ├── request conversion ├── retry logic ├── error classification ├── circuit breaker └── tool-call recovery Google integration ├── request conversion ├── retry logic ├── error classification ├── circuit breaker └── tool-call recovery I did not want to build resilience three times. I wanted to define the recovery policy once and apply it across every provider supported by the application. That became llm-api-resilience , a Python library for retries, ordered failover, circuit breakers, and checkpoint recovery across multiple LLM providers. Quick start pip install llm-api-resilience Once the provider adapters are configured, an ordered fallback plan takes only a few lines: from llm_api_resilience import RecoveryPlan , ResilientLLM , Route llm = ResilientLLM ( RecoveryPlan ( [ Route ( " openai-primary " , openai_adapter ), Route ( " anthropic-backup " , anthropic_adapter ), Route ( " google-last-resort " , google_adapter ), ] ) ) response = llm . chat ( [{ " role " : " user " , " content " : " Explain circuit breakers briefly. " }] ) print ( response . selected_route ) print ( response . content ) GitHub: llm-api-resilience Provider adapter: llm-api-adapter The library is built on top of llm-api-adapter , which provides one interface for OpenAI, Anthropic, and Google. Because provider-specific differences are handled by the adapter, the resilience layer can operate on a shared contract inst

Troops received an email informing them that they were rapidly depleting their AI tokens.

Originally published at tengli.dev Your MCP server can be 100% spec-compliant and still be unusable by an agent. The Model Context Protocol spec tells you how to transport tools: JSON-RPC framing, capability negotiation, schema shapes. It says nothing about whether a model can actually use what you serve — whether it picks the right tool out of your catalog, fills the arguments correctly, or burns 8k tokens parsing your schemas on every single request. I integrate first- and third-party MCP connectors into a production AI agent for a living, and I kept seeing the same failure: servers that pass every compliance check, yet the model calls the wrong tool, hallucinates arguments, or ignores the tool entirely. The problems were never in the protocol layer. They were in the parts no one lints: descriptions, naming, schema design. So I wrote mcpgrade — a Lighthouse-style scorecard for MCP servers. One command, no API key, report in seconds: npx mcpgrade --stdio "npx -y your-mcp-server" Then I pointed it at 36 popular servers. It did not go great. The results Full sortable table: https://tengli.dev/mcp-leaderboard.html . The short version (static analysis, point-in-time snapshot; servers marked (archived) are unmaintained reference implementations, included because they're still widely installed and copied): Top of the class (A): brave-search (archived) , exa, google-maps (archived) , slack (archived) , perplexity-ask, @shopify/dev-mcp, @apify/actors-mcp-server, airbnb, figma-developer-mcp, tavily, gitlab (archived) , elastic, shrimp-task-manager, and more — 15 of 36. Bottom of the class (D/F), 11 of 36 — and it's not hobby projects: MongoDB's official server (66, with 66 errors), Notion's official server (62), Airtable (69, 66 errors), todoist-mcp-server (67, 110 errors ), GitHub's archived reference server (67, 44 errors), and firecrawl-mcp at the very bottom (57, 134 errors ). Two more servers (Stripe, Supabase) couldn't be scanned with dummy credentials and were exclud

Every team shipping an agent has the same meeting. Someone asks "is it ready?" and the room splits. One person saw a great demo. Another watched it invent a refund policy an hour ago. The argument runs in circles because nobody agreed what "ready" means, so the loudest opinion wins and the agent ships on a vibe. Making an AI agent production-ready is not a moment of confidence. It is a bar you write down before you build, then measure against. This post is about that bar: why agents need a different one than the services you already ship, and how to define it so "is it ready?" becomes a number instead of an argument. Why "production-ready" breaks for agents For a normal service, "production-ready" is settled. Correct output for valid input, handles errors, meets a latency target, has tests and a rollback. You know the shape of done. An agent breaks three of those assumptions at once: It is non-deterministic. The same input can produce different output, so "correct" becomes "acceptably right, often enough." Its failure surface is open-ended. A function fails in ways you enumerated; an agent fails in ways you never imagined, because it composes language, tools, and judgment on the fly. Its worst case is not a 500 error. It is a confident wrong answer that looks right, which is far more expensive than a crash, because a crash at least tells you it failed. So the honest question is not "is the agent correct." It is "is the agent acceptably wrong, safely, within budget, and repeatably enough to trust." That question has four parts, and each is a line on your bar. The four lines of the bar Write these down before you build. If you cannot fill them in, you do not have a spec, you have a wish. Task success. On a fixed set of real tasks , not the happy-path demo, what fraction must the agent complete correctly? Pick the number. 85 percent means one in seven users gets a wrong answer. Acceptable for this job, or fireable? Decide on purpose. Failure acceptability. Not all wron

Who the F*** Am I? Hi, I'm Daniel Flores. A software developer with, I believe, seven years of professional experience. It's been a wild ride — at least the past three years. I was one of the first to actually try GitHub Copilot during its preview, probably around 2021 or 2022. I was completely amazed by it, but it was definitely very, very rough around the edges. Nobody knows me, though. I never intended to become a public figure or one of those guys who "knows where AI should go." That's not me, and that's okay. I've been watching this new paradigm evolve — and devolve — from the sidelines. What I Think About LLMs and "AI" I've been watching videos about this topic for years. Evolution simulators, learning algorithms, a model trained to play hide and seek — I was completely baffled when I realized that video came from OpenAI itself. I'm not an expert in how to build models or LLMs. I have no PhD. I've just been doing my own thing for years, watching which tools get adopted and which ones suck. And I have to say this: the industry doesn't know what the hell is happening. Neither do I. Nobody knows, and that's what I hate the most. LLMs are extremely useful. They can save you a lot of hours of work. But that's only half the story. So What's the Issue? Almost everyone — paid AI shills, mostly — is telling you: "YOU'RE GETTING BEHIND IF YOU DON'T USE THESE TOOLS!!" They're trying to push the entire industry into a fear-of-missing-out state. Let me share my personal experience about this completely inconsequential fear. If you're already experienced enough — if you already know what an LLM is and can prompt it to do or refactor something — you're not missing out. That's all there is to it. I'm going to explain what I mean. The exact same issues I had with the old GitHub Copilot — I don't even know what model it ran, probably a customized GPT — are still true today with the latest frontier models. They all hallucinate. They all seem to kind of understand what they're do