标签：#llm

The Problem LLM evaluation tools like Patronus AI excel at hallucination detection, toxicity checks, and semantic relevance. But they don't catch the structural failures: A JSON response missing required fields A function call with malformed parameters Output that violates schema constraints Latency budget overruns silently degrading UX Cost explosions from runaway token usage These aren't hallucinations. They're verification failures. The Solution correctover-patronus is an adapter that runs Correctover's 87 deterministic verification rules as native Patronus evaluators. Every verdict comes with a recomputable proof hash — meaning you can verify the verifier. pip install correctover-patronus The 6 Dimensions Dimension What It Checks Example Structure Output format validity JSON parses correctly Schema Field presence & types Required fields exist Identity Semantic relevance to input Response addresses the question Integrity Forbidden pattern absence No Tracebacks or error messages Latency Response time budget Under 30s threshold Cost Token usage budget Under 10k token limit Usage Full 6-Dimension Verification from correctover_patronus import CorrectoverEvaluator , CorrectoverConfig config = CorrectoverConfig ( min_confidence = 0.7 , latency_rules = { " max_ms " : 5000 }, cost_rules = { " max_tokens " : 4000 } ) evaluator = CorrectoverEvaluator ( config = config ) result = evaluator . evaluate ( task_input = " Summarize this article... " , task_output = " The article discusses... " , task_context = { " source " : " article " , " word_count " : 1500 } ) print ( f " Overall: { result . score : . 2 f } ( { ' PASS ' if result . pass_ else ' FAIL ' } ) " ) print ( f " Proof hash: { result . metadata [ ' proof_hash ' ] } " ) for dim , info in result . metadata [ ' dimensions ' ]. items (): print ( f " { dim } : { info [ ' status ' ] } (score= { info [ ' score ' ] : . 2 f } ) " ) Individual Dimensions from correctover_patronus import correctover_structure , correctover_inte

Of all the things involved in turning a woman's life into a 60-second reel, I assumed that the writing would be the easy part. Surely telling a good story in 150 words is exactly what a large language model should be good at; yet it proved surprisingly difficult. Draft after draft suffered from common AI weaknesses: a tendency to use hyperbole and inspirational language, to generalize, follow generic founder arcs ("built in a basement"), and focus on morbid details. (For this effort, I was using Sonnet 4.6, which I found to be better — more grounded, more true to facts, less inventive — than Opus 4.7 at creating longer bios.) Getting to something publishable took a significant amount of work. That said, the human in this story also found writing good short story arcs surprisingly challenging, so the effort spent on getting the system to do it decently was well worth it. Here's some context, then what I did. I've been publishing short biographies of notable women for a while now, on a website called Tycoona . Why notable women? Because in each field there are so many women who contributed so much and are little known. I've never understood why Corita Kent , the pop-artist nun, isn't as famous as Andy Warhol, why Hetty Green , the Gilded Age value investor called both the "Witch of Wall Street" and the "Queen of Wall Street," disappears into history behind Benjamin Graham and his protege Warren Buffett. The problem I faced is that no one was seeing my bios, so I decided to make short videos or reels in hopes of increasing my reach. Creating the technical infrastructure for the reels on top of my existing system was fun and relatively straightforward. Each reel consists of a series of "beats," and Remotion turns those beats into a vertical reel, complete with on-screen text and royalty-free images & attributions pulled from Wikimedia, Flickr, or Library of Congress. For content, the beat generator relies on the knowledge base of validated facts that my system creates f

You've got a folder of a few hundred screenshots and you want the text out of each one. Or you want to generate a batch of images for a side project. Or you just want to drop a single "summarize this" call into a script you're writing on a Sunday afternoon. So you open the pricing page for the official API, do the math on per-token billing plus setting up keys and a payment method, and it's hard to justify, because the exact same model will do the exact same thing for free in a browser tab. There are really two ways to get a model like ChatGPT or Gemini to do work for you. The web UI is free, or already covered by a subscription you're paying for anyway, but you drive it by hand. The API is scriptable, but you pay by the token. Most of the time that trade-off is fine. But for a whole category of work like hobby projects, throwaway scripts, research, or anything that doesn't need production-grade reliability, you're stuck picking between "free but manual" and "automated but paid." Which raises the obvious question: why not automate the free web UI? It's just a webpage. You open it, type in the box, click send. It turns out that hides a few fiddly problems, which I ran into enough times that I eventually built a small library for them. In this article we'll work through what it takes to automate these UIs, and at the end I'll show how little code it comes down to. 1. What it takes to drive a chat UI A single round trip with ChatGPT or Gemini breaks down into four jobs: Get your text into the input box Optionally attach a file Wait for the model to finish answering And read the answer back out. Every one of these is harder than it sounds, because the page is a modern single-page app that was never built to be driven by a script. We'll use Selenium with undetected-chromedriver, and for now assume the browser is already open (we'll get to launching it in the next section). To keep the code readable I'll show whichever of the two platforms makes each problem clearest, and

I had a bug last month that took most of a Saturday to find. A support bot we shipped started promising refund timelines that didn't match policy. Customer complaints, frantic Slack messages, the usual. The prompt had changed three weeks earlier. Nobody could remember why. Git blame pointed to a one-line edit inside a 200-line SYSTEM_PROMPT constant. No PR description, no diff worth reading. That's when I knew I'd been writing prompts wrong for the last two years. PromptOT - Prompt Management Platform Compose prompts from typed blocks, version safely, and deliver to your apps via API. The prompt management platform built for AI engineering teams. promptot.com Prompts are code, but we treat them like Notion docs A typical system prompt for anything useful crams five things into one string: You are a friendly support agent for Acme. Use this knowledge: {{kb}}. Follow escalation rules. Never share internal ticket IDs. Reply in plain text, two to four paragraphs. That's a role, context, instructions, guardrails, and an output format all jammed together. When the PM wants to soften the tone, they're editing the same string an engineer uses to update the knowledge base. When security adds a guardrail, it lands inches from the response format. One bad edit and every reply ships broken. We wouldn't write code this way. So why are prompts always a 200-line const somewhere in lib/ ? What I built PromptOT is a prompt management platform. The core idea is small: typed blocks instead of flat strings. You break a prompt into pieces. Each piece has a type — role, context, instructions, guardrails, output_format, custom. Each one is independently editable, can be toggled on or off, and has its own version history. The compiler joins them into a single prompt string at delivery time. Block 1 — role : " You are a support agent for Acme..." Block 2 — context : " Knowledge base: {{kb}}..." Block 3 — instructions : " 1. Acknowledge the issue..." Block 4 — guardrails : " Never share inte

IndexCache: Killing the Indexer's O(NL²) Bottleneck in DeepSeek Sparse Attention Notes from my notebook on GLM-5.2 / DeepSeek Sparse Attention (DSA), reconstructed from the IndexCache paper (Bai, Dong et al., Tsinghua + Z.ai, 2026) — the mechanism behind GLM-5.2's "IndexShare." 1. Why this exists — the bottleneck nobody talks about DSA's whole pitch is: don't do full O(L²) attention, instead let a cheap lightning indexer look at all preceding tokens and pick the top-k (k=2048) that actually matter, then do real attention only on those. That drops core attention from O(L²) → O(Lk). Great — except I missed this the first time I read DSA: the indexer itself is still O(L²) . It has to score every preceding token against the query to decide who's in the top-k. So across N layers you've traded one O(L²) cost for N separate O(L²) costs — total O(NL²). At long context this indexer becomes the dominant cost, not the attention it was supposed to fix. Adding the indexer is "DSA on steroids" because it kills DSA's one real bottleneck (full attention) — but in doing so, it grows its own. The indexer is cheap per-FLOP (few heads, low-rank, FP8) but it still runs at every single layer. The fix the paper proposes isn't a smarter indexer — it's don't run it every layer at all. 2. The core insight: adjacent layers pick almost the same tokens If you measure pairwise overlap between the top-k token sets selected by each layer's indexer, adjacent layers share 70–100% of their picks. The heatmap even shows block structure — clusters of layers (e.g. layers 3–5, 17–30, etc.) that all converge on roughly the same "important" tokens. So most of the O(NL²) indexer cost is redundant computation of the same answer. This motivates IndexCache : split the N layers into two roles — F (Full) layers — run their own indexer, compute fresh top-k, cache it. S (Shared) layers — skip the indexer entirely, just reuse the nearest preceding F layer's cached top-k. The first layer is always F (has to seed the

Wix-owned vibe coding platform Base44 has started rolling out its own AI model — with hopes that it will eventually outperform frontier models.

How I ended up running a local LLM to generate my git commit messages

How to Create an AI Agent: A Production Walkthrough The first agent I shipped to production failed at 3am on a Sunday. It looped on a tool call, burned through $40 in tokens before my budget alarm fired, and left a half-written draft in the database with no way to resume. That night taught me more about agent design than any framework tutorial. Since then I have built a pattern I trust enough to leave running unattended for weeks at BizFlowAI, where agents research, write, optimize and publish content without me touching them. This is that pattern, stripped down to what actually matters. Start with the job spec, not the framework Before you pick LangGraph, CrewAI, or roll your own, write the agent's job spec like you would for a junior engineer. One paragraph. What it owns, what it must never do, what "done" looks like, and which signals tell you it failed. Here is the spec for one of my production agents: The Topic Researcher owns generating a ranked list of 20 content topics per site per week. It reads from keyword_pool and search_console_perf , writes to topic_queue . It must never publish, never call paid APIs more than 8 times per run, and must finish in under 6 minutes. Done = 20 topics with score >= 0.6 and zero duplicates against the last 90 days. Failure signal = empty queue after a run, or any topic flagged by the dedupe check. If you cannot write this paragraph, do not build the agent. You will end up with a "do everything" prompt that hallucinates its way through ambiguous tasks. The job spec becomes your evaluation rubric later, so write it carefully. Rule of thumb I use : if the spec needs more than 5 tools or more than 3 decision branches, it is two agents, not one. Design the tools before you write the prompt Most agent failures I have debugged were not prompt failures. They were tool failures. The model called a tool with wrong arguments, the tool returned a 4MB JSON blob, or two tools had overlapping responsibilities and the model picked the wrong

The AI Implementation Process I Use With Every Client Most AI projects do not fail at the model. They fail in the six weeks before anyone writes a prompt, and in the six weeks after the demo lands in a Slack channel and nobody knows who owns it. I have run enough of these now (from one-off automations to multi-agent content systems running unattended) that the process has converged into something stable. This is the version I actually use. It has five phases: scoping, POC, integration, evaluation, operations. Each phase has an exit criterion. If we cannot meet the exit criterion, we do not move forward. That single rule has saved more projects than any clever architecture choice. Phase 1: Scoping (1 to 2 weeks, fixed price) Scoping ends with a written document that names the workflow being automated, the system of record it touches, the success metric in hours or dollars, the data we have access to, and the smallest possible first slice. No model is chosen yet. No code is written. If we cannot produce that document, the engagement stops here and the client keeps the document. The hardest part of scoping is resisting the urge to solve the interesting problem. Clients almost always describe the AI-shaped fantasy ("an agent that handles all support tickets") when the real opportunity is narrower and uglier ("triage tier-1 tickets that mention billing, route to the right queue, draft a reply for human approval"). The narrower version ships. The fantasy does not. I run scoping as three sessions: Workflow walkthrough. Someone who actually does the work shows me their screen for an hour. I record it. I take timestamps. The point is to find the moments where a human is doing pattern matching that an LLM can do, and to find the moments where they are doing judgment that an LLM should not do. Data audit. Where does the input live? Where does the output need to go? What is the auth story? If the data is locked inside a SaaS product with no API and no export, that is the projec

I spend a lot of time in the AI space -- reading papers, building things, talking to engineers who are actually shipping. And there is a gap between what the demos show and what production systems actually look like that nobody is being fully honest about. So here is my honest take on where things actually are. The Problem With How We Talk About AI Agents Everyone is calling everything an "agent" right now. A function that calls a tool? Agent. A chatbot with memory? Agent. A script with a loop? Agent. This dilution is not just semantic. It is causing real engineering mistakes. When you do not have a precise definition for what you are building, you end up over-engineering simple pipelines and under-engineering genuinely complex ones. I have seen teams spend weeks adding "agentic" orchestration to workflows that would have been fine as a single well-structured prompt. Here is the definition I keep coming back to: an agent is a system that has an objective, not just an instruction. It decides what to do next. It handles failure. It knows when it is done. Everything else is just a fancy function call. 🟢 If your system needs a human to tell it each step, it is not an agent. It is a chat interface. 🔵 If your system can recover from a failed tool call and try a different approach, you are getting somewhere. ✅ If your system can decompose a goal into subtasks and delegate them, that is the real thing. What Is Actually Happening in Production Right Now The honest picture from teams I follow and talk to: Most real agent deployments are narrow. They do one thing well. Customer support triage. Document extraction. Code review on a specific codebase. They are not general-purpose reasoning engines. They are purpose-built pipelines with some intelligence in the decision layer. The teams getting good results are not chasing the latest model release. They are obsessing over: ☑️ Tool design -- what can the agent actually call, and how clean is the interface ☑️ Failure handling -- wh

Key takeaways Summarizing conversation history can reduce costs by up to 60%. Implementing an effective summarization algorithm is key to efficiency. Balancing detail and brevity in summaries is crucial for context. Optimized context windows lead to faster response times and lower latency. The problem Startups leveraging large language models (LLMs) often face significant costs associated with managing context windows during conversations. Each token processed incurs a cost, and as conversations grow, replaying entire histories can lead to runaway expenses. Founders and engineers encounter this issue particularly during customer support interactions or chatbots, where lengthy dialogues require constant context retention, drastically inflating operational costs. What we found Our research indicates that instead of replaying the entire conversation history, summarizing the dialogue can maintain context while drastically reducing token usage. By distilling key points and intents into a concise summary, we can effectively minimize the number of tokens processed, leading to major cost savings without sacrificing the quality of interaction. This non-obvious insight repositions how we approach conversation management in LLMs. How to implement it Start by selecting a summarization algorithm suitable for your use case. Techniques like extractive summarization (e.g., using TextRank) can identify and retain essential sentences from conversations, while abstractive methods (e.g., fine-tuning a transformer model) rephrase the content. Next, integrate this summarization step into your workflow: after each interaction, generate a summary that captures the main points. Ensure that the summary is stored and utilized as context for subsequent interactions, replacing the need for the entire conversation history. Monitor token usage before and after implementation to quantify cost savings. How this makes life easier By summarizing conversation history, startups can see a reduction in c

Would you trust a sci-fi author to program critical AI systems for humanity? No? Yet, that's what we've been doing. Years ago, I remember hearing the argument: "Why don't we just prompt LLMs with Asimov's three laws of robotics ?" It sounds elegant. The laws were designed to constrain artificial minds. Why not use them? Because the model has already read every story where they fail. LLMs are statistical engines designed to autocomplete text. Imagine a story that starts like this: Once upon a time, there was a good little robot who followed the 3 laws of robotics to the letter. Now take human literature and complete the story. Does it end well? ‹ › (function() { var container = document.currentScript.closest('.ltag-slides--carousel'); var track = container.querySelector('.ltag-slides__track'); var slides = track.querySelectorAll('.ltag-slide'); var prevBtn = container.querySelector('.ltag-slides__nav--prev'); var nextBtn = container.querySelector('.ltag-slides__nav--next'); var dotsContainer = container.querySelector('.ltag-slides__dots'); var current = 0; var total = slides.length; for (var i = 0; i < total; i++) { var dot = document.createElement('button'); dot.className = 'ltag-slides__dot' + (i === 0 ? ' ltag-slides__dot--active' : ''); dot.setAttribute('aria-label', 'Go to slide ' + (i + 1)); dot.dataset.index = i; dot.addEventListener('click', function() { goTo(parseInt(this.dataset.index)); }); dotsContainer.appendChild(dot); } function goTo(index) { current = ((index % total) + total) % total; track.style.transform = 'translateX(-' + (current * 100) + '%)'; var dots = dotsContainer.querySelectorAll('.ltag-slides__dot'); for (var i = 0; i < dots.length; i++) { dots[i].classList.toggle('ltag-slides__dot--active', i === current); } } prevBtn.addEventListener('click', function() { goTo(current - 1); }); nextBtn.addEventListener('click', function() { goTo(current + 1); }); })(); It doesn't. Because the entire body of fiction built around those laws exists to explo

Your AI coding agent writes something that looks right. It compiles in your head. Then you notice it called user.getProfileById() — a method that doesn't exist anywhere in your codebase. You didn't ask it to make that up. It invented it confidently, in the middle of otherwise-fine code. And that's the worst kind of wrong: not obviously broken, just quietly incorrect in a way you have to catch. If you've run Claude Code, Cursor, or any agent on a real repo, you know this one. Here's why it happens — and why the obvious fix doesn't work. The fix everyone tries first (and why it fails) You reword the prompt. You add "Don't make up functions." It behaves… for one file. Then it does it again. So you add "Only use methods that exist in the provided code." Better for a bit. Then two more sentences — and now your prompt is fifteen rules long and it still invents a method the moment the task gets complex. Here's the part nobody tells you: rewording treats a structural problem as a vocabulary problem. A prompt isn't a contract the model reads once and obeys. It's something the model has to hold in working memory while it reasons about your actual task. A flat list of fifteen rules is unholdable. As the work gets harder, the model spends its attention on the code and quietly drops whichever rule wasn't front-of-mind. "Don't invent methods" is usually rule #11. Under load, it falls out. You can't out-word that. A sixteenth rule just gives it one more thing to drop. The actual cause: shape, not wording The agent invents a method because nothing in the prompt's structure requires it to check. You told it what not to do. You never changed what it actually does, step by step. So stop forbidding the bad thing. Remove the opportunity for it. Instead of a rule it has to remember, make grounding a required step it has to perform. Before — a pile of rules:You are an expert engineer. Write clean code. Follow our conventions. Don't make up functions. Only use methods that exist. Handle er

Last night I was deep in a build session with an AI assistant. We picked it back up tonight. At some point I mentioned it had been a day and a half since we last spoke — and the model had no idea. None. As far as it knew, it was still the previous session. The gap was invisible to it. That tiny moment is one of the most underrated problems in AI systems right now. So let's talk about it. The model doesn't know what time it is An LLM gets a rough sense of "now" at the start of a conversation — a single timestamp, handed to it once. That's why it can greet you with "good morning." But that stamp is frozen. It doesn't update as the conversation runs, and it definitely doesn't travel into the next conversation. Each session starts cold. On its own, that's a curiosity. It becomes a real problem the moment the model reasons over retrieved context — search results, documents, database rows, another agent's output. Staleness is invisible Here's the dangerous part. When a model reads a retrieved document, that document usually carries no trustworthy signal about when it was true . So the model treats it as present-tense. It produces a confident answer from six-month-old data with nothing flagging that the data is old. A few places this bites: Pricing — quoting a number that changed last quarter. Availability — "in stock" from a cached page. Compliance — citing a policy that was superseded. People — stating someone's job title from two years ago. For a human reader, a slightly stale search result is fine — you see the date and judge for yourself. For an LLM, the staleness is silent. The wrong answer looks exactly like a right one. Why "just add a clock" doesn't fix it The instinct is: give the model the current time. But knowing it's 9 PM doesn't help if the document you're citing went stale in 2023 and nothing told you. The missing piece isn't the model's clock — it's the context's freshness . Two different things: What time is it now? — easy, a now() call solves it. How old

The biggest challenge wasn't choosing a language model or designing prompts—it was managing context over time. Once an application grows beyond isolated conversations, memory becomes just as important as reasoning. An assistant that remembers previous architectural decisions, coding preferences, and project history can contribute much more effectively than one that starts from scratch every session. Runtime intelligence proved to be equally important. Not every request deserves the same computational resources. Routing tasks based on complexity, enforcing execution budgets, and maintaining an audit trail make AI systems more predictable and practical for real-world development. DevPilot AI brings these ideas together by combining Google Gemini for reasoning, Hindsight for persistent memory, and cascadeflow for runtime intelligence. While the project will continue to evolve, building it reinforced one idea above all else: the future of AI applications isn't just about generating better responses. It's about building systems that can remember, adapt, and make better decisions over time. If you're interested in the architecture or would like to explore the project further, you can find the source code here: GitHub: https://github.com/siddharthg-7/DevPilot-Ai- I'm always interested in feedback and discussions around persistent memory, runtime intelligence, and AI engineering. If you've explored similar ideas or approached these challenges differently, I'd love to hear your perspective.

Apple's newest on-device model carries about 20 billion parameters, and on any given request it fires maybe one to four billion of them. That gap — 20B stored, roughly 3B running — is the whole story of 2026. The model that now ships inside the latest iPhone is no longer a shrunken, lobotomized cousin of the cloud model. It's a different kind of object: large in flash, small in motion, and it never phones home. For three years the on-device pitch was mostly aspirational. Demos ran, latency was rough, quality trailed the API by a generation, and every serious AI feature still resolved to a per-token bill in someone's datacenter. In mid-2026 that stopped being true. Two releases — Apple's third-generation Foundation Models at WWDC on June 8, and Google's Gemma 4 family on April 2 — quietly moved the floor. Genuinely useful agents now run on hardware you already own, offline, for free. The economics nobody priced in Forget benchmarks for a second; the load-bearing fact here is accounting. When the model lives in the cloud, every inference is a metered event — input tokens, output tokens, a line item that scales linearly with usage and explodes the moment you wrap the model in an agent loop. Agentic workloads are the worst case for the token meter: a single "go do this task" can fan out into dozens of model calls as the agent plans, calls tools, retries, and re-reads its own output. The bill grows with your ambition. Move the model onto the device and the marginal cost of an inference is approximately $0 . No API key, no rate limit, no usage dashboard. You paid for the silicon once; every token after that is free in the only sense a product manager cares about — it doesn't show up on a monthly invoice that grows with your success. That single change rewrites which features are worth building. A background task that re-summarizes your inbox every five minutes is insane on a per-token plan and trivial on-device. So is an agent that quietly loops a hundred times to get one

Give a capable coding agent a real, multi-week project and watch what breaks. It isn't intelligence. It's continuity. Every session starts cold or half-remembered. Context windows fill up and compact. The thread of what we decided, what's true, and what's done starts to fray. Over a long horizon the same failures keep coming back: the agent claims state it never actually verified, reports something done with no proof it ran, quietly drifts from the project's conventions, and loses hard-won context that lived only in the last session's head. Bigger context windows don't fix this. They just postpone it. We've been building a real product with a forgetful agent as the primary engineer for weeks now, and the thing that made it work isn't a clever prompt. It's a simple recognition: transmission across a stateless agent needs two channels, and most setups only build one. The first channel is structure, which is discipline made un-forgettable. These are the deterministic guards that run whether or not the agent remembers to care: a pre-commit check that refuses a "done" without a real, verifiable artifact; a hook that blocks a sloppy search and points at the right tool instead; a scan that won't let a secret reach a transcript; a status snapshot generated from the repository's actual state instead of hand-kept prose that quietly goes stale. The rule we keep coming back to is that a guard is the system's discipline made un-forgettable. A fresh session follows the hard-won lessons without having to remember them, because the structure enforces them at the moment of action. The second channel is soul, which is the why, kept human. This is the short orientation a session reads before it starts working: who to be, what the work is ultimately for, and why the discipline exists at all. It's the difference between an agent that complies and one that understands. Structure can transmit the what, but only prose can transmit the why. And the why matters, because an agent that only fo

Why Knowledge Alone Doesn't Create Judgment Karpathy's LLM Wiki is brilliant. You dump raw material in, an LLM extracts concepts and links them together, and you get a personal knowledge base that actually works. I built one. 100+ pages. It's great. But I hit a wall that made me rethink everything. The Wall I asked my AI to act as a programming tutor. It could recite every concept perfectly. Student: "I don't understand Promises." AI: "A Promise is an object representing the eventual completion or failure of an asynchronous operation..." Wrong answer. The right answer was: "Do you understand callbacks first? What about synchronous execution? What have you tried so far?" The AI had knowledge. It had zero judgment. And then I realized why: every single page in my wiki was the same type of knowledge. One Type vs Four LLM Wiki 1.0 stores declarative knowledge — facts, definitions, summaries. Things that answer "What is this?" But think about what makes a human expert different from a textbook: A great programming mentor doesn't just know what Promises are. They know why you teach callback → Promise → async/await in that exact order — and never the reverse. That's not a fact. It's a reasoning path. A master astrologer doesn't just know what each star represents. They know why you check 命宮 first, then 三方四正, when to prioritize 格局, when a palace is a consequence rather than a cause. That's not a fact either. It's a decision sequence. And here's the kicker: even knowing the reasoning path isn't enough. We annotated Anderson's (1972) Socratic tutoring dialogues — full 41-turn and 30-turn conversations, labeling every decision point. Knowing the 23 Socratic rules (the reasoning path) is one thing. Reading a complete dialogue — watching the expert set a trap, wait 15 seconds in silence, break their own rules when the student gets frustrated — is something else entirely. Knowing the recipe ≠ having watched the chef cook. And there's still one more type. Student says: "I have no

I gave GLM-4.5-Air (106B, open weights) 12 coding tasks through opencode on my RTX 3090. It scored 0% — never edited a single file. Same model, same GPU, same tasks, but driven by a ~150-line LangGraph agent instead: 93% . The model was never the problem. The orchestrator was. Here's the benchmark — including the part nobody else measures, the electricity cost per correct task . Setup RTX 3090 (24 GB) + 128 GB RAM , models via ollama , Q4 quants, temp 0.2 5 recent open models × 2 orchestrators (opencode vs custom LangGraph ReAct with ollama-native tool-calling) 17 graded tasks (12 coding in Python/JS/C++ + 5 general-agent) with hidden unit tests Every run priced in GPU watts via my open-source homelab-monitor Results Model tok/s opencode adh. LangGraph adh. LangGraph coding LangGraph general Qwen3-Coder 30B-A3B 130 92% 100% 100% 100% GLM-4.5-Air 106B 5.7 0% 100% 89% 100% Devstral Small 24B 49 8% 53% 8% 40% Seed-OSS 36B 9.5 0% 7% 0% 20% DeepSeek-R1-Distill 32B 6.7 0% 0% 0% 0% Tool-adherence = % of tasks where the model actually called a tool instead of just printing code in chat. It was the master variable. (GLM's headline "93%" is its blended score across all 17 tasks: 89% coding + 100% general.) Three takeaways The framework can matter more than the model. opencode sends a frontier-shaped system prompt + 12 tools over its OpenAI-compat path; most local models fall back to chatting. Native tool-calling through a lean agent fixes that — GLM went 0% → 93%. (Qwen3-Coder is the exception: it's tuned for agentic tool use and aces opencode out of the box.) Acting ≠ solving. LangGraph made Devstral act (8% → 53% adherence) but not solve (coding stayed 8%). The framework decides whether a model acts; the model decides whether it's right. The wattmeter ranks honestly. Qwen solved tasks at ~0.0005 BGN each; the models that scored zero still burned 10–30× more energy for nothing. On a home rig, the cheapest model is the fast, correct one — and MoE (Qwen activates ~3B of 30B pe

Introduction Speculative decoding is one of those techniques that has been "almost ready for production" for the better part of three years. A small draft model proposes tokens; a larger target model verifies them in a single forward pass. In theory, you get 2–4× throughput. In practice, the draft model has to be cheap, fast, and good enough at mimicking the target's distribution, which is a much harder combination than it sounds. Yesterday, a new paper from DeepSeek quietly climbed to the top of Hacker News (714+ points, 290+ comments at the time of writing). It's called DSpark , and it reframes speculative decoding in a way that looks like it could finally make the technique drop-in rather than bolt-on. The paper is here: github.com/deepseek-ai/DeepSpec/blob/main/DSpark_paper.pdf The Core Idea Instead of training a separate, smaller draft model from scratch (the classic approach), DSpark grafts the speculative head directly onto the target model. The intuition is simple: if the target model already knows which tokens are likely to follow, why not reuse its own intermediate representations rather than maintaining a parallel network? From the discussion on HN, this approach has a concrete architectural benefit — it reduces layer duplication that you'd otherwise have to maintain with a standalone draft model. In the DeepSeek experiments, the technique was applied on top of Step and Qwen 3.6 , which are themselves MTP-capable. How It Fits With MTP One of the more interesting practical points raised by HN commenters: DSpark is complementary to Multi-Token Prediction (MTP) , not a replacement for it. MTP — where the model predicts several future tokens at every step using auxiliary heads — has already been shown to give 50–100% speedups on hardware like the NVIDIA DGX Spark. DSpark adds another layer on top: even with MTP, the validation step is still a single forward pass through the main model, and the speculative tokens that get accepted come "for free." A useful men