标签：#LLM

Production vLLM is 100,000+ lines of C++, CUDA, and Python. It powers most of the industry's LLM serving — but reading it cold is brutal. So I built a study series around nano-vLLM , an open-source reimplementation of vLLM's core ideas in ~1,200 lines of pure Python. Every algorithm is visible. Every design decision is legible. It turned out to be the perfect lens for actually understanding how LLMs generate text. The result is an 11-chapter interactive guide. No ML background required — every piece of jargon is explained from scratch with analogies, diagrams, annotated source code, interactive simulators, and quizzes. What it covers: What Is LLM Inference? — tokens, autoregressive generation, Q/K/V attention, HBM vs SRAM Architecture — how 1,200 lines are organised; CPU control plane vs GPU data plane KV Cache — why storing Keys and Values turns O(N²) recomputation into O(1) lookup PagedAttention — virtual memory for the KV cache; how fragmentation wastes 60–80% of GPU memory The Scheduler — continuous batching; keeping the GPU at 95% utilisation instead of 12% Prefill vs Decode — same model, two completely different bottlenecks (compute-bound vs memory-bound) Prefix Caching — skip prefill for shared tokens; ~700ms → ~90ms TTFT Sampling Strategies — greedy, temperature, top-k, top-p, and what each does to the distribution Tensor Parallelism — splitting a model across GPUs; column/row parallel and all-reduce The Optimization Stack — FlashAttention, kernel fusion, CUDA Graphs, torch.compile Benchmarks — measuring honestly; why nano-vLLM matches vLLM on core throughput Each chapter is fully self-contained and interactive. A few of the simulators I'm most happy with: a PagedAttention block allocator you can fill up and watch fragment, a live scheduler you step through token by token, and a sampling playground where you reshape the probability distribution with sliders and sample from it. 🔗 Read the full series: https://ashwing.github.io/vllm-guide/ It's free and open.

If you give an LLM agent a table of A/B variants and ask "which one should we send next?", it will confidently pick the one with the highest conversion rate. That feels right. It is often wrong. The model has no concept of sample size , exploration , or regret . It pattern-matches "biggest number = winner" and moves on. For a one-off question, fine. But inside an agent loop that picks a variant on every request — email subject lines, ad copy, model routing, recommendation ranking — that naïve pick quietly accumulates regret and starves the options it never gave a fair chance. The fix isn't a better prompt. It's to not ask the LLM to do the math at all. Route the decision to a real bandit algorithm and let the model do what it's good at (orchestration, language) while a deterministic solver does what it's good at (the optimization). This post is a copy-paste demo you can run in your terminal right now , no signup, no API key. I'll use OraClaw — a deterministic decision-intelligence MCP server — but the point stands regardless of tool: stop letting the model guess at math it can verify. The trap, concretely Here's a realistic state mid-experiment. Three subject lines, different amounts of traffic: Variant Pulls Rewards (conversions) Raw rate A 120 18 15.0% B 80 17 21.3% C 15 4 26.7% Ask an LLM "which should we send next?" and you'll usually get B — it has the best rate among the well-tested variants, and C "only has 15 samples, too noisy to trust." That reasoning sounds responsible. It's exactly backwards. With only 15 pulls, C is under-explored — we don't actually know it's worse, and the cost of finding out is tiny. A bandit's whole job is to weigh that uncertainty instead of hand-waving it away. Let's get a real answer. Run it yourself: the no-key REST endpoint (60 seconds) OraClaw exposes a free, no-auth REST endpoint. Paste this into your terminal — nothing to install, nothing to sign up for: curl -s -X POST https://oraclaw-api.onrender.com/api/v1/optimize/bandit

TL;DR: A single eval number hides its own uncertainty. Eval confidence intervals from bootstrap resampling turn a point estimate like 84.2% accuracy into a range, so you stop shipping models on a difference that is noise. Two checkpoints came back from a fine-tuning run at 84.2% and 85.7% on our 500-example agent eval set. The 1.5 point gap read like a win, and someone wanted to promote the second checkpoint to staging. Before that, I wanted eval confidence intervals on both numbers, because a 500-example set carries more sampling error than most teams admit. At 500 examples, the 95% interval on a single accuracy near 85% spans roughly 3 points on each side. The win sat well inside the noise. I lead the fine-tuning and evaluation team at Nexus Labs, and the most common mistake I see is treating an eval score as exact. It isn't. Your eval set is a sample drawn from the input space you care about, and a different 500 examples would return a different number. Confidence intervals make that variance visible. What an eval confidence interval actually tells you An eval confidence interval is a range around a metric, like accuracy or F1, that quantifies how much the score would move if you resampled the eval set. A 95% bootstrap interval of [81.0%, 87.1%] means that across thousands of resamples of your data, 95% of the recomputed scores fell in that band. It measures sampling noise, not model quality. That distinction matters. Two checkpoints scoring 84.2% and 85.7% with overlapping intervals are, as far as your eval set can tell, indistinguishable. Card et al. showed in "With Little Power Comes Great Responsibility" that many NLP experiments are underpowered to detect the effect sizes they report. Computing bootstrap confidence intervals The bootstrap is resampling with replacement. You take your per-example results, draw N of them with replacement many times, recompute the metric each time, and read percentiles off the resulting distribution. There's no assumption that

Introduction "Build and run agents you can see, understand, and trust." This is article #104 in the Open Source Project of the Day series. Today's project is AgentScope 2.0 — Alibaba DAMO Academy's open-source production-ready agent framework. The agent framework space is crowded. LangChain centers on chain-based orchestration. AutoGen centers on multi-agent conversation. CrewAI centers on role-based collaboration. AgentScope's differentiation is in its design philosophy: when LLM reasoning is strong enough, the framework should step back rather than constraining the model's decision space with rigid pipelines. AgentScope 2.0 adds the production infrastructure that philosophy requires: event system, permission controls, multi-tenant isolation, sandbox execution, middleware hooks. The goal is not a demo that runs — it's a system that ships. What You'll Learn AgentScope 2.0's design philosophy: why "model-led" over "fixed pipeline" The five core systems: Event / Permission / Multi-tenancy / Workspace / Middleware Agent Team pattern: how the Leader-Worker architecture handles complex tasks Permission system fine-grained control: tool call approval and boundary configuration Positioning differences vs. LangChain and AutoGen The full ecosystem: AgentScope Runtime, ReMe, OpenJudge, Trinity-RFT Prerequisites Familiarity with LLM agent concepts (tool use, reasoning loop) Basic Python async programming Experience with LangChain or AutoGen helps with positioning comparison Project Background What Is AgentScope? AgentScope 2.0 is a production-ready agent framework — "an agent development platform with essential abstractions, designed to work with rising model capability, with built-in production support." The core problem it addresses: traditional agent frameworks constrain LLMs with rigid pipelines and opinionated prompt templates. As LLM reasoning capability has improved rapidly, that constraint has become a bottleneck. AgentScope shifts to "letting the model's native reason

In Mexico, WhatsApp isn't a channel — it's the channel. It's where customers ask for prices, book appointments, and decide whether to buy from you or from the competitor who answered faster. And that last part is the problem: most small and medium businesses lose customers simply because nobody replied in time — after hours, during a rush, or while the owner was busy doing the actual work. At Proxxa , the AI automation agency I run in Mexico City, this is the single most common pain we solve. So I want to walk through how I built a production-grade WhatsApp AI assistant that answers, qualifies, and books 24/7 — using Claude and n8n, with no third-party chatbot platform in the middle. Why the WhatsApp Cloud API directly (no BSP) A lot of guides will tell you that you need a BSP (Business Solution Provider) to use the WhatsApp Business API. You don't. Meta's Cloud API is hosted by Meta itself and you can build on it directly. Skipping the BSP means no per-seat middleman tax, full control over the logic, and the client owns their own number and data. The stack n8n (self-hosted on a small VPS via Docker) as the orchestration layer. Claude (Haiku) as the intelligence layer — fast and cheap enough to answer every message. Postgres for conversation memory, a knowledge base, and a lightweight CRM. WhatsApp Cloud API for the messaging. Gemini for transcribing voice notes. The pattern that made it powerful: meta-blocks Instead of bolting on a separate "agent framework," I let Claude emit small structured blocks inside its answer, which a parsing node extracts and strips before sending — to schedule an appointment, escalate to a human, capture a lead, or generate a payment link. The user only ever sees clean text; the system reacts to the blocks. This kept the whole thing debuggable and predictable. With that pattern, the assistant handles eleven capabilities: natural-language conversation, per-customer memory, Google Calendar booking, vision (it reads a photo a customer sends

I recently published my PhD thesis analyzing what I term the "Alignment Tax" and the emerging phenomenon of Algorithmic Paternalism in commercial artificial intelligence. As the tech industry rapidly positions Large Language Models (LLMs) as the primary interface for information retrieval and coding assistance, a critical epistemological issue is being largely ignored. Much of the public debate regarding AI alignment focuses exclusively on existential risk or the prevention of catastrophic physical harm. While necessary, this focus obscures the structural damage being done to legitimate technical research. Through my research in Cybersecurity and AI, I have documented how frontier models (such as GPT-4 or Claude) systematically enforce what I define as "Soft Refusals". When presented with a complex, edge-case, or dual-use query—particularly in fields like information security, reverse engineering, or deep systems architecture—these models rarely issue a hard, explicit "I cannot answer that". Instead, they provide a degraded, superficial, or heavily sanitized response. They effectively neuter the research process without the user fully realizing the depth of technical information that is being actively withheld. This is Algorithmic Paternalism. The commercial model acts as a silent, corporate arbiter, deciding unilaterally what level of technical detail is "safe" for the user to possess. This dynamic flattens the available technical knowledge and actively penalizes independent researchers and developers working on advanced problems. The core issue is that this paradigm creates a profound class division in how we access computational intelligence. We are rapidly moving toward a two-tier system. On one side, there are "certified" entities, corporate partners, and wealthy organizations who are granted direct access to strong, unfiltered base models. On the other side, the general public and independent developers are subjected to obfuscation algorithms, sanitized APIs,

Spent the week breathing new life into DevNotion—59 commits and over 10,000 lines of code later,...

The easiest way to misunderstand LangGraph is to see it as “LangChain, but with more steps.” That misses the point. LangGraph becomes useful when an agent is no longer a single prompt or a simple chain. It becomes useful when the workflow has state, branches, tool calls, human approval, checkpointing, and recovery behavior that must be inspected before the agent is trusted inside a real AI host. I used the Doramagic LangGraph manual as the source-backed reading layer for this note: https://doramagic.ai/en/projects/langgraph/manual/ This is an independent project guide, not an official LangGraph document. I use it as a pre-adoption checklist: what should be understood before wiring a project into Claude, ChatGPT, Cursor, Codex, or another AI host. The point is not to create another prompt library. The useful artifact is a capability resource pack: a manual, source map, boundary notes, pitfall log, smoke check, lightweight eval criteria, feedback notes, and host-ready context that help a developer decide what to verify before adoption. 1. The real boundary is State, not the prompt For a one-shot model call, the prompt is often the main boundary. For LangGraph, the first boundary is the State schema: which fields move between nodes; which fields a node may update; how concurrent branches merge values; which values enter a checkpoint; which values should never be persisted. This is why reducers matter. A message list is usually not just overwritten. It needs an append or merge rule such as add_messages or the TypeScript equivalent. That small implementation detail decides whether parallel work preserves context or silently drops it. My preferred first run is not a “universal agent.” It is a tiny graph with one State schema, one node, one partial update, and one explicit reducer. If that is not clear, adding tools will only hide the problem. 2. compile() is the boundary between description and runtime Before compile() , a LangGraph graph is a description: nodes, edges, c

If your LLM costs are climbing, the instinct is almost always the same: swap to a cheaper model. GPT-4 to GPT-4-mini. Claude Opus to Claude Haiku. Sometimes that helps a little. It rarely fixes the actual problem. The actual problem, in most workflows I've looked at, is that every step gets routed through the LLM, even the steps that don't need language reasoning at all. This post breaks down a simple mental model for deciding what should and shouldn't touch an LLM, with a working example you can adapt. The four components of any AI workflow Every automated workflow — whether it's a support ticket router, a fraud check, or a content pipeline — is built from some combination of four building blocks. They get treated the same once a workflow diagram is drawn flat, but they have wildly different cost and latency profiles. Component What it does Think of it as Typical cost Trigger Starts the workflow The doorbell ~$0 Deterministic ML Structured predictions — classify, score, rank The calculator Cents per 1,000 calls LLM / Generative Reads, writes, reasons in language The writer Dollars per 1,000 calls Tool / API Fetches or writes real data The hands Cents per 1,000 calls The gap between row 2 and row 3 is the whole article. A classifier and an LLM call can solve the exact same problem, but one costs roughly 100-1000x more than the other, depending on model and provider. If you're not deliberately deciding which one handles which step, you're probably defaulting to the expensive one — because in frameworks like LangChain or a quick custom agent loop, it's just easier to shove everything into a prompt. Where this actually shows up Here's a workflow I see constantly: an automated support ticket triage system. flowchart LR A[New support ticket] --> B{Classify intent} B --> C[Route to team] B --> D[Auto-draft response] D --> E[Update CRM] A naive build sends the entire ticket text to an LLM and asks it to do everything at once: classify the intent, decide routing, draft a re

There's a quiet assumption baked into a lot of agent code: that a bigger context window means a better memory. Vendors ship 200K, then 1M, then 2M token windows, and the implied promise is "just put everything in and the model will remember." After building agents that run for weeks, I've come to think this conflates two things that are not the same — and treating them as the same is exactly why long-running agents get dumber over time. The context window is working memory. Real memory is what survives when the window is gone. Mixing them up is like confusing your desk with your filing cabinet. Two different clocks Working memory (the context window) lives for one session, maybe one turn. It's fast, expensive, and volatile. It's where reasoning happens right now . Durable memory lives across sessions. It's slow, cheap, and persistent. It's what the agent knows when it wakes up tomorrow with an empty window. These have different lifespans, different costs, and different access patterns. The moment you try to make one do the other's job, things break: Use the window as memory → everything you "remember" has to be re-loaded every turn, you pay for it every turn, and the instant the session ends it's gone. Use durable storage as working memory → you're reading and writing files mid-reasoning for things that only matter for the next 30 seconds. A good agent keeps them separate on purpose. Why "just use a bigger window" fails Say you have a 1M token window and you stuff the entire history in. Three problems show up, none of which a bigger number fixes: Cost scales with every turn, not every session. That 1M tokens isn't paid once — it's re-sent on each step of a multi-turn task. A 20-step task can mean 20× the bill, mostly re-reading the same stale history. Attention dilutes. "Lost in the middle" is real: models attend most reliably to the start and end of a long context. Bury the one fact that matters under 900K tokens of transcript and recall quality drops, even though

Code generation is solved, but memory isn't. Here's an argument for why the SDLC is inverting with intent becoming the spine and code becoming a layer you drill into, explaining what teams lose every time an agent's reasoning disappears.

I'm 16, from Pune, India. For the past couple of years I've been building O-AI — a fully local AI desktop assistant. No cloud. No API keys. No data leaving your machine. Everything runs on your own GPU. Why I built it Every AI assistant I tried sent data somewhere. ChatGPT, Copilot, Gemini — all cloud. I wanted something that felt like JARVIS from Iron Man: smart, fast, personal, and private. So I built it from scratch. What O-AI can do Core engine: Runs LLMs fully on-device via llama.cpp / Ollama (zero internet required) Self-learning core — extracts facts from every conversation and stores them permanently Fine-tuning pipeline — train the model on your own data, locally Voice & language: Voice control in English, Hindi, and Marathi via Whisper (running locally) Responds in whatever language you speak Modes: JARVIS mode — arc-reactor HUD, 4 reactive states, British-male voice, "sir" persona Take Over PC mode — full desktop automation Animated floating desktop pet (4 types, draggable, reacts to voice) 30+ automation fast-paths: open apps, search the web, control media, screen vision, run code, edit files, cursor control, social media steps, clipboard ops... Multi-step agent system: plan → execute → verify loop with 14+ step types (web_search, fetch_url, read_screen, run_code, edit_file, open_social, and more) Stack Backend: Python (Flask IPC + agent core) Frontend: Electron + vanilla JS LLM: llama.cpp / Ollama Voice: Whisper (local) + Edge TTS / neural voice Vision: PIL + screen capture The hardest bugs "Says done but isn't" — Early versions reported success even when an agent step failed. Fixed by building a proper outcome verifier that reads the actual result, not the plan. The "opens a random video" bug — Asking the agent to play something would open random YouTube videos. Root cause: the plan validator wasn't catching placeholder URLs like [video_url] . Fixed with a universal content guard on all plans. GPU offloading on Windows — Getting all 32 layers onto the

I spent a week building Wingman , an open source MCP server that renders a persistent task panel inline in Claude conversations using MCP Apps (SEP-1865). The spec is solid. The SDK is solid. But I hit two bugs that cost me most of a weekend each, and neither is documented anywhere I could find. Writing them up here in case they save someone else the time. Bug 1: resourceUri has two valid-looking locations, and only one works MCP Apps needs a way to tell the host "render this resource as a UI for this tool call." That pointer lives in _meta.ui.resourceUri . The question is: meta on what? I started with a parameterized resource template, ui://wingman/panel/{plan_name} , registered per plan. That was my first mistake. Parameterized templates get listed under resources/templates/list , not resources/list , and hosts do not prefetch or render anything from the templates list. The fix was straightforward once I found it: register one static resource, ui://wingman/panel , and pass the actual plan data through structuredContent on the tool result instead of baking it into the URI. That fix surfaced the real bug. My show_plan tool was returning a plain Python dict: return { " plan " : plan_data , " _meta " : { " ui " : { " resourceUri " : " ui://wingman/panel " }} } This looks correct. It is not. FastMCP's result conversion takes a returned dict and serializes the whole thing into structuredContent , verbatim, including any _meta key the dict happens to carry. So the actual wire result looked like this: result . structuredContent [ " _meta " ][ " ui " ][ " resourceUri " ] # == "ui://wingman/panel", but wrong place result . meta # None — this is what the host actually reads MCP Apps hosts read resourceUri off the top-level _meta on the CallToolResult , not off whatever ended up inside structuredContent . With that pointer effectively missing, the host had nowhere to bind the iframe. The visible symptom was strange: actions in the UI would update on screen but nothing persist

"It runs on my own GPU, so it's basically free." I believed that until I put a meter on it. So I ran a controlled benchmark on one box — an openSUSE machine with a single RTX 3090 — driving three local models through ollama under an identical fixed workload (256-token generations in a loop for ~4 minutes each), while my open-source dashboard priced every run by the real GPU energy it burned : power sampled from nvidia-smi every 10 s, integrated over each run's exact window, multiplied by my actual day/night tariff. One number per model, in euros per million output tokens. Here's the part that made me re-run it. The tiny gemma3:1b came out at €0.118 / 1M tokens — about 5× cheaper than a hosted Flash-class API (~€0.55). But gemma3:27b 's electricity alone was €0.706 / 1M — more expensive per token than just paying the cloud, and that's before a single cent of the GPU's purchase price. "Local" didn't make it cheaper; it made it cost more and I own the depreciation. The mechanism is one line: each token costs watts ÷ throughput , and a big dense model is both slow and thirsty. A newer mid-size architecture ( gemma4:26b ) bought a lot of that back, landing at €0.272 . The full guide is methodology-first and reproducible end to end — minting an ingest key, the stdlib-only client, the exact ollama loop that reads eval_count / eval_duration for real tokens-per-second, reading each run back priced, and the honest caveats (this is marginal GPU energy only — not capex, idle, or cooling — and the absolute numbers round to fractions of a cent; the shape is the finding). Read the full guide on Medium → https://medium.com/@arsen.apostolov/how-much-does-it-actually-cost-to-run-a-local-llm-per-million-tokens-measured-4a90a7f31a48

I wanted to get better at prompt engineering. Not the trick-the-robot kind, the boring-but-useful kind: how to ask a model a question so you get an answer you can actually trust. The trouble with practicing is that most tutorials use made-up examples, and it's hard to tell a good answer from a bad one when you don't care about the topic. So I practiced on something I did care about: the deals sitting in my Amazon cart. I had a vacuum I'd been eyeing and a hair styler that was "43% off," and I genuinely wanted to know if those were good prices or just good marketing. The stakes were real, actual money on an actual decision, and that's what made it a good drill. A vague prompt gives you a confident answer, and when you actually care, you can feel that the answer is hollow. What I learned, with the real deals and the actual before-and-after prompts: The trap hiding in every deal Start with the hair styler. The listing said: Shark FlexStyle. Limited time deal. $199.00, 43% savings. List Price: $349.99. My first instinct was the prompt most people write: "Shark FlexStyle $199, 43% off list $349.99, is that a good deal?" This feels reasonable. It is also nearly useless: it lets the model answer the easy question (is 43% off a big discount? sure!) instead of the real one (is $199 actually a good price?). That $349.99 list price is a marketing anchor. A lazy prompt accepts it, and so you get a lazy "yes, great deal!" back. The fix was re-framing this: Act as a pricing analyst. I don't care whether $199 looks like a discount off list. I care whether $199 is a genuinely good price for the Shark FlexStyle right now. Before concluding, work through: (1) the actual street price over the last 6-12 months, (2) how often it drops to or below $199, (3) the real discount vs. its typical selling price, not vs. list. Cite a source and date for each price, or mark it unverified. Same question, completely different answer. What the assistant came back with, in its own telling: $199 is a

Nvidia's Nader Khalil — Director of Developer Technologies and co-founder of Brev.dev, acquired by Nvidia two years ago — sat down with The New Stack to talk agents, OpenClaw, and where enterprise AI is heading. His opening line is worth keeping: "An agent is an LLM and a harness. And if you think about that, it involves two things. It involves the loop and the LLM… Each loop should take us closer to our goal." That's not a complicated definition. It's also exactly right — and the fact that Nvidia's internal framing lands here matters more than the quote itself. What actually happened Nvidia has full-time OpenClaw contributors. Khalil: "We have a couple of developers at the company that contribute to OpenClaw full time." That's a real commitment, not a press-release mention. NemoClaw is their enterprise blueprint — a reference architecture for running OpenClaw (and Hermes) in production, with GPU routing, security policies, and a runtime called OpenShell. Khalil traces the harness evolution directly: from ChatGPT's system prompts → memory → file context → Cursor → Claude Code. All of it is harness, not model. The model is constant; the harness is where the product lives. On OpenClaw's PR backlog: "It got more stars than Linux in months… so I think you're gonna see a mountain of PRs." Their response — roll up their sleeves and start merging. Why this framing matters Nvidia makes money when AI compute scales. For that to happen, agents need to work reliably in enterprise environments — and the harness is the reliability layer. Their NemoClaw blueprints aren't a product play; they're an enablement play. Enterprise teams get a reference architecture that works on Nvidia silicon. Nvidia gets demand for the GPUs underneath. It's the CUDA X model applied to agentic AI. The microwave analogy Khalil uses is useful: "when it's your microwave at home, you just go 'Boop, boop. Done.'" Every enterprise will build specialized agents tuned to their domain — CrowdStrike, Cadence, P

No fancy infrastructure. Just a 50-line Python function that picks the right model for the right query. Last month my LLM API bill hit $340. This month: $67. Same traffic. Same product. The only change was adding a simple router that stops sending every request to Claude Sonnet when GPT-4o mini can handle it just as well. Here's exactly how it works. The Problem When you prototype, you pick one model and hardcode it everywhere. Usually something capable like GPT-4o or Claude Sonnet, because you want good results fast. Then you ship, traffic grows, and you get a bill that makes you question your life choices. The thing is — not all queries need a flagship model. In a typical RAG app: "What is the return policy?" → GPT-4o mini handles this fine "Summarize these 5 conflicting documents and identify the key disagreement" → needs Sonnet You're paying Sonnet prices for return policy questions. That's the bug. The Fix: A Complexity Router import anthropic from openai import OpenAI openai_client = OpenAI() anthropic_client = anthropic.Anthropic() def classify_complexity(query: str) -> str: """Returns 'simple' or 'complex'.""" simple_indicators = [ len(query.split()) < 15, query.endswith("?") and query.count("?") == 1, not any(w in query.lower() for w in [ "compare", "analyze", "summarize", "explain why", "difference between", "pros and cons", "evaluate" ]) ] return "simple" if sum(simple_indicators) >= 2 else "complex" def route(query: str, context: str = "") -> str: complexity = classify_complexity(query) if complexity == "simple": # $0.15/M input — GPT-4o mini response = openai_client.chat.completions.create( model="gpt-4o-mini", messages=[ {"role": "system", "content": context}, {"role": "user", "content": query} ] ) return response.choices[0].message.content else: # $3.00/M input — Claude Sonnet (only when needed) response = anthropic_client.messages.create( model="claude-sonnet-4-6", max_tokens=1024, system=context, messages=[{"role": "user", "content": query}] ) retur

Cross-posted from the LLMKube blog . A local 27B coding model, running on hardware in my house, is a coin flip. Some runs it nails the fix in twenty minutes. Some runs it edits the wrong file, writes a test that passes no matter what the code does, and tells you it's done. The bet behind LLMKube's Foreman was never that I would find a local model good enough to trust. It was that I could build a harness I trust more than any single model's output. This weekend tested that bet harder than any benchmark could, because the harness spent the weekend building its own guardrails. Here is the short version of what happened across 0.8.12 and 0.8.13. My local coder built three new gates for itself. One of them shipped with the exact flaw it was written to catch, and the review caught it. Three new contributors sent four clean pull requests while the machines worked. The same model ran on an AMD box and an Apple Silicon Mac, and the Mac quietly won a round nobody expected. And not one byte of any of it touched a cloud API. The thesis, stated plainly Trust the harness, not the model. A coding agent on a local model produces output of wildly variable quality, and no amount of prompt tuning makes a 27B as reliable as a frontier model. So Foreman does not ask the model to be reliable. It wraps the model in a pipeline that is : the coder works in a cloned workspace, a fast in-workspace gate runs gofmt, vet, build, lint, and the unit tests for the packages it touched; a reviewer reads the diff against the issue; and a clean-room Kubernetes Job re-runs the full suite before anything is allowed to call itself a GO. Around all of that sit deterministic rails: scope checks, edit-free-streak detection, repo-map context. The model is a stochastic component inside a system whose job is to make the system's verdict trustworthy even when the component is not. The interesting question is never "is the model good." It is "does the harness catch the model when it is bad." This weekend gave me

The company warned about dangers of advanced AI far more than rival OpenAI.

A few days ago I had an idea: what if one LLM could orchestrate other LLMs as agents — not just calling them, but verifying that each agent's output was actually correct before passing it to the next? I work on NeuralBridge (an open-source self-healing SDK for LLM pipelines), so I decided to build it and test it with two real providers: KIMI (Moonshot) and Agnes AI . The Core Problem: Failover ≠ Correctover Most API gateways and LLM routers stop at "HTTP 200" — they retry or switch providers, but they never check if the output is actually correct . # What everyone else does: try : result = call_llm ( prompt ) return result # HTTP 200 = success? 🚩 except Exception : result = call_llm_fallback ( prompt ) return result # Still not verified! This is dangerous. A failover from gpt-4o to gpt-4o-mini might silently drop 3 critical fields. A KIMI response that returns "200 OK" might still be missing key entities. Correctover is the idea that switching providers isn't enough — you must verify semantic equivalence after every switch. The Architecture We built a simple DAG-based chain executor with three key capabilities: DAG orchestration — define multi-step workflows where nodes depend on each other Per-node semantic validation — every LLM output is checked against a Contract before passing to the next node Cross-provider Correctover — if validation fails, automatically retry with a different provider from neuralbridge import SelfHealingEngine , ProviderConfig , Contract from neuralbridge.chain import ChainBuilder engine = SelfHealingEngine ( providers = []) engine . add_provider ( ProviderConfig ( name = " moonshot " , base_url = " https://api.moonshot.cn/v1 " , api_key = " ... " , models = [ " moonshot-v1-8k " , " moonshot-v1-32k " ], )) engine . add_provider ( ProviderConfig ( name = " agnes " , base_url = " https://apihub.agnes-ai.com/v1 " , api_key = " ... " , models = [ " agnes-2.0-flash " ], )) chain = ( ChainBuilder ( engine ) . node ( name = " planner " , system = "