标签：#LLM

Running a 70B parameter model to summarize a 200-word email is wasteful. Running a 3B model to review production code is reckless. Most systems live somewhere in between — and that's where model routing comes in. It matches task complexity to model capability. The tradeoffs are real, but the savings are too. The routing problem People usually start with one model and stick with it. That works until you notice the cost, or the latency, or both. The alternative is building a router — something that decides which model handles which request. Four strategies work in practice: Capability-based — route by what the model can do Cost-aware — route by what you're willing to spend Latency-aware — route by how fast you need it Hybrid — combine them Each optimizes something different. Picking one is usually a decision about what hurts most. Capability-based routing The simplest approach. Classify the task, send it to the model that handles it. Task Model size Examples Classification, tagging 1-3B Qwen2.5-1.5B, Gemma-2-2B Summarization, extraction 3-7B Qwen2.5-7B, Llama-3.1-8B Code generation 7-14B Qwen2.5-Coder-7B, DeepSeek-Coder-V2 Complex reasoning 14-32B Qwen2.5-32B, Llama-3.1-70B Creative writing, analysis 32B+ Qwen2.5-72B, Claude, GPT-4 If the task doesn't need the bigger model, don't use it. A 1.5B model handles sentiment classification fine. It just won't write a coherent essay. Implementation is straightforward: ROUTING_RULES = { " classify " : { " model " : " qwen2.5-1.5b " , " max_tokens " : 100 }, " summarize " : { " model " : " qwen2.5-7b " , " max_tokens " : 500 }, " code_review " : { " model " : " qwen2.5-coder-7b " , " max_tokens " : 2000 }, " reason " : { " model " : " qwen2.5-32b " , " max_tokens " : 4000 }, " creative " : { " model " : " claude-sonnet-4 " , " max_tokens " : 8000 }, } def route_request ( task_type : str ) -> dict : return ROUTING_RULES . get ( task_type , ROUTING_RULES [ " reason " ]) The catch is classification itself. If you get the task type

Single-model systems are simple. Multi-model systems are powerful. The challenge isn't choosing models — it's designing the architecture that orchestrates them. A multi-model system isn't about having more models. It's about having the right model for the right task at the right time. Architecture patterns Five patterns cover most use cases: Pattern Complexity When to use Tradeoff Single Model Lowest Prototyping, simple tasks Limited capability Sequential Low Multi-step workflows Higher latency Parallel Medium Independent tasks Higher cost Hierarchical High Complex reasoning Complex orchestration Ensemble Highest Critical decisions Highest cost Pick the simplest one that works. Complexity is real, and it compounds. Sequential architecture Process tasks through a chain of models, each specializing in a step. Pattern 1: Pipeline Pipeline pattern — each model's output feeds the next: class ModelPipeline : def __init__ ( self ): self . models = [ { " model " : " qwen2.5-1.5b " , " task " : " classify " }, { " model " : " qwen2.5-7b " , " task " : " extract " }, { " model " : " qwen2.5-32b " , " task " : " reason " }, ] def process ( self , input : str ) -> str : current = input for model_config in self . models : current = self . call_model ( model_config [ " model " ], self . create_prompt ( model_config [ " task " ], current ) ) return current Latency adds up. Three models in sequence means three times the latency. Only use this when each step actually needs a different model. Pattern 2: Router Router pattern — classify the task, route to the specialist: class ModelRouter : def __init__ ( self ): self . classifier = " qwen2.5-1.5b " self . specialists = { " code " : " qwen2.5-coder-7b " , " math " : " qwen2.5-32b " , " creative " : " claude-sonnet-4 " , " general " : " qwen2.5-7b " , } def route ( self , prompt : str ) -> str : task_type = self . classify ( prompt ) model = self . specialists . get ( task_type , self . specialists [ " general " ]) return self . call_m

Building a prototype of an AI agent is fun. Building a production-ready agent is a nightmare. In a perfect world, your agent always gets the perfect context, the API never fails, and the model never gets "lazy." But in the real world, transient errors are a constant, and models love to take shortcuts. If you aren't testing your agent against the messy reality of production, you’re setting yourself up for failure. This is where our Agent Profiler comes in. We’ve designed it to be an "adversity sandbox." It doesn’t just ask your agent a question; it challenges it. We inject transient runtime errors, introduce "lazy-agent traps" that force the model to stay focused, and validate structural AST matches to ensure the agent is actually outputting what it claims to output. It’s an active testing loop designed to stress-test your agent’s self-recovery mechanics. If your agent can’t handle a little chaos in the test suite, it certainly won’t survive your users.

What if you could build a production-ready AI agent workflow in 10 lines of YAML — and have it handle retries, observability, and multi-model routing out of the box? Dify is an open-source LLM app development platform with 145,764 GitHub stars, 22,915 forks, and 460+ contributors. It just shipped v1.14.2 (May 2026) with security hardening, agent groundwork, and workflow reliability improvements. Yet most teams only use it as a no-code chatbot builder — completely missing the infrastructure underneath. In 2026, AI workflows have moved from "prompt and pray" to orchestrated multi-step pipelines with memory, tool calling, and observability. Dify sits at the center of this shift, combining visual workflow design, RAG pipelines, agent capabilities, and LLMOps in a single platform that runs on your own infrastructure. Here are 5 hidden uses of Dify that most teams never discover. Hidden Use #1: Visual Workflow as Code — Export, Version Control, and Replay What most people do: Build workflows in the Dify web UI, click "Run," and hope for the best. When something breaks, they debug by clicking through nodes manually. The hidden trick: Every workflow in Dify can be exported as YAML. You can version-control it in Git, diff changes between deployments, and replay any historical execution step-by-step using the built-in tracing API. # dify-workflow.yaml — a production RAG + agent pipeline app : name : " customer-support-agent" mode : " workflow" version : " 1.14.2" nodes : - id : " start" type : " start" variables : - name : " user_query" type : " string" required : true - id : " retriever" type : " knowledge-retrieval" dataset_ids : [ " faq-dataset-v3" ] top_k : 5 score_threshold : 0.7 depends_on : [ " start" ] - id : " llm-agent" type : " llm" model : " gpt-4o" prompt_template : | Context: {{ retriever.documents }} Question: {{ start.user_query }} Answer concisely using only the context above. depends_on : [ " retriever" ] - id : " output" type : " end" output : " {{ llm-agen

如果你能用 10 行 YAML 构建一个生产级的 AI Agent 工作流——并且自带重试、可观测性和多模型路由——你会怎么做？ Dify 是一个开源的 LLM 应用开发平台，拥有 145,764 个 GitHub Stars、22,915 个 Fork、460 多位贡献者。它刚刚发布了 v1.14.2（2026 年 5 月），包含安全加固、Agent 基础架构和工作流可靠性改进。但大多数团队只把它当作无代码聊天机器人构建器——完全忽略了底层的基础设施能力。 2026 年，AI 工作流已经从"写个 prompt 然后祈祷"进化到了具备记忆、工具调用和可观测性的多步骤编排管道。Dify 正处于这个转变的中心，将可视化工作流设计、RAG 管道、Agent 能力和 LLMOps 整合在一个可以部署在你自己基础设施上的平台中。 以下是大多数人从未发现的 Dify 的 5 个隐藏用法。 隐藏用法 #1：可视化工作流即代码——导出、版本控制与回放 大多数人的做法： 在 Dify 网页 UI 中构建工作流，点击"运行"，然后祈祷一切正常。出了问题就手动点击每个节点来调试。 隐藏技巧： Dify 中的每个工作流都可以导出为 YAML。你可以在 Git 中进行版本控制，对比不同部署之间的差异，并使用内置的追踪 API 逐步回放任何历史执行。 # dify-workflow.yaml — 一个生产级 RAG + Agent 管道 app : name : " customer-support-agent" mode : " workflow" version : " 1.14.2" nodes : - id : " start" type : " start" variables : - name : " user_query" type : " string" required : true - id : " retriever" type : " knowledge-retrieval" dataset_ids : [ " faq-dataset-v3" ] top_k : 5 score_threshold : 0.7 depends_on : [ " start" ] - id : " llm-agent" type : " llm" model : " gpt-4o" prompt_template : | 上下文：{{ retriever.documents }} 问题：{{ start.user_query }} 请仅使用以上上下文简洁回答。 depends_on : [ " retriever" ] - id : " output" type : " end" output : " {{ llm-agent.text }}" depends_on : [ " llm-agent" ] tracing : enabled : true backend : " langfuse" # 或 opik、arize-phoenix sample_rate : 1.0 效果： 你的整个 AI 管道变成了基础设施即代码。你可以 CI 测试工作流变更、回滚到历史版本、审计每次执行追踪——就像管理 Terraform 或 Kubernetes 清单一样。 数据来源： Dify GitHub 145,764 Stars、22,915 Forks（GitHub API，langgenius/dify，2026-06-19 推送）。最新版本 v1.14.2（2026-05-19）包含工作流可靠性修复。460+ 贡献者（GitHub API 确认）。 隐藏用法 #2：多模型路由与自动降级 大多数人的做法： 选一个模型（通常是 GPT-4），硬编码到每个工作流节点中。当这个模型出现故障或限流时，整个管道就崩溃了。 隐藏技巧： Dify 的模型配置支持提供程序级别的自动降级链。你可以配置主模型、备用模型，甚至为非关键路径配置第三级廉价模型——所有这些都无需修改工作流逻辑。 # dify_model_config.py — 通过 Dify API 配置多模型路由 import requests DIFY_API_KEY = " your-api-key " DIFY_BASE = " https://your-dify-instance.com/v1 " def configure_model_fallback (): """ 为生产弹性配置 3 层模型降级链。 """ # 主模型：GPT-4o，高质量推理 # 备用 1：Claude 3.5 Sonnet（不同提供程序，同等级别） # 备用 2：GPT-4o-mini（廉价、快速，简单步骤足够用） config

Architecture Overview Clioloop's Agentic Fusion is not just "run the same prompt 5 times and pick the best." It's a structured pipeline where different models play different roles, with strict security boundaries between them. The Pipeline Step 1: Planning Phase When you run /fusion , up to 5 planner models are dispatched in parallel. Each planner: Receives your original prompt and context Has read-only tool access — they can search the web, read files, but never modify anything Proposes an approach (not an answer — an approach) The planners might suggest different strategies: Planner A: "Search the web for similar problems, then write a script" Planner B: "Read the existing codebase first, then modify in place" Planner C: "Break it into subtasks and use the Kanban system" Step 2: Execution Phase Your main model takes the planners' proposals and does the actual work: Full tool access (file editing, shell, web, browser, image gen, etc.) Fully visible — you watch every file edit, every command, every web search in real time Not a black box — you can intervene at any time This is the key difference from "ensemble" approaches: the main model does real work with real tools, not just text generation. Step 3: Review Phase Up to 5 reviewer models critique the draft: Read-only access — they can see what the main model produced, including generated images They check for errors, suggest improvements, flag problems Each reviewer works independently Step 4: Verdict Loop The draft is revised based on reviewer feedback: If reviewers find issues, the main model gets the feedback and revises The loop continues until reviewers approve You get the final, reviewed answer Step 5: Fusion Everything combines into one answer that has already passed independent review. Security Model The safety comes from schema-level restrictions : Role Can Read Can Write Can Execute Planners ✅ Files, web, images ❌ Nothing ❌ Nothing Main Model ✅ Everything ✅ Files ✅ Commands Reviewers ✅ Draft, files, image

The Problem Most AI assistants give you one model's answer. If it's wrong, you catch it or you don't. If you use a cheap model, quality drops. If you use a frontier model, you pay frontier prices for everything — even a simple file rename. What is Agentic Fusion? When you run /fusion , a panel of models collaborates on your task: Planners (up to 5): Read-only models that research and propose routes in parallel. They figure out the best approach but can't touch your files or run commands. Main model : Your chosen model does the actual work — full tool access, fully visible. You watch every step. Not a black box. Reviewers (up to 5): Read-only models that critique the draft. They can see images the main model generated. They check for errors, suggest fixes, flag issues. Verdict loop : The draft is revised until reviewers approve. The answer you get has already passed independent review. Fusion : Everything combines into one reviewed, approved answer. The quality comes from synthesis — not from running the same job 5 times. Cheap open models combine into something that rivals a frontier model at a fraction of the cost. Safety by Construction Planners and reviewers are read-only at the schema level. They can research and critique, but they can never touch your files or execute commands. Only your main model has tool access, and you watch it work live. Beyond Fusion Clioloop is also: Self-improving : Keeps MEMORY.md and USER.md , updated automatically Autonomous : Set a standing goal with /goal and it loops until done Everywhere : Terminal, desktop app, web dashboard, Telegram, Slack, Discord, WhatsApp Multi-agent Kanban : Break big work into tasks with worker agents Tools : File editing, shell, web search, browser, image/video gen, TTS, MCP Scheduled jobs : Run on cron for automated workflows Open-source : Self-host everything, own your data The Omni Loop Portal One OAuth login gives you access to 300+ models. No API keys. An OpenAI-compatible proxy means any tool works

AI-disclosure: AI-assisted draft, human-reviewed. The demo numbers are the verbatim stdout of a deterministic, stdlib-only Python script included in full below — re-run it and you get the same bytes. The attempt counts in that script are a SYNTHETIC fixture I chose to exercise the accounting mechanism, calibrated to the retry skew I see in my own scraper logs (run counts from my Apify history). It is NOT a benchmark of any named vendor's API or prices. The one external claim (the cost-per-successful-task formula) is attributed and linked. The cheaper API was 2.5x cheaper per call. The monthly bill came in higher anyway. Not by a rounding error. The "cheap" option cost 1.63x more per successful task than the one with the bigger sticker price. Same workload. The price page never showed me that number, because the price page doesn't know your success rate. You do — after you've already paid. This is the arithmetic the per-call price hides. And it's a decision you make before you spend, not a cap you bolt on after. TL;DR You compare two API tiers by per-call (or per-token) price and pick the cheaper one. That ranking can be wrong. You pay for every attempt — success or fail. The denominator that pays the bills is successful tasks , not calls. True cost = price_per_attempt × attempts ÷ successes . A cheap tier with a low success rate burns its discount on retries. In the run below: cheap tier $0.0020 /call but $0.0096 /success; robust tier $0.0050 /call but $0.0059 /success. The sticker winner loses . For anyone choosing an API, model, or tier for an agent: log attempts and successes for a week, divide, then decide. A 70-line script is at the bottom — drop in your numbers. The price page is a sticker, not a bill Here's the trap, stated plainly. The number on the pricing page is per call . The number on your invoice is per call too — but the value you got is per successful task . Those are different denominators, and the gap between them is exactly the work that failed. E

Originally published at kunalganglani.com — read it there for inline code, hero image, and live links. Generative AI vs agentic AI vs AI agents. Three terms, used interchangeably by people who should know better, burning engineering budgets across the industry in 2026. Generative AI refers to models that produce new content — text, images, code — from a prompt. AI agents are software systems that wrap those models with planning, memory, and tool use to pursue goals autonomously. Agentic AI is the broader paradigm: orchestrated systems of agents, workflows, and decision-making that operate with minimal human oversight. Getting these distinctions wrong doesn't just lose you a Twitter argument. It determines whether your production system costs $500/month or $50,000. Every quarter, someone on a leadership team says "we need to go agentic." What they usually mean is one of three completely different things. And the architecture you pick for each one has wildly different implications for cost, latency, reliability, and maintenance burden. I've watched teams burn entire quarters building autonomous agent systems when a well-tuned prompt engineering pipeline would have shipped in a week. That's not a hypothetical. I watched it happen twice in 2025. This post cuts through the buzzword soup. I'll define all three paradigms with concrete technical distinctions, show you how they map to real production architectures, and give you a decision framework for picking the right one. What Is Generative AI? The Engine, Not the Vehicle Generative AI is the foundation layer. It's a large language model (or image model, or audio model) that takes an input and produces new output. GPT-4, Claude, Gemini, Llama — these are all generative AI. You send a prompt, you get a completion. That's it. The critical thing to understand: generative AI is stateless by default . Each API call is independent. The model doesn't remember what you asked five minutes ago. It doesn't plan a sequence of steps.

At least one malware developer is adding text about nuclear and biological weapons to their spyware, in an effort to stop automatic AI analysis. Details : The _index.js payload begins with a large JavaScript block comment containing fake system instructions and policy-triggering content. Because it is inside a comment, it does not affect JavaScript execution. The runtime skips it. The real malware begins after the comment with a try{eval(…)} wrapper around a large character-code array and a ROT-style substitution function. This header appears designed for AI-mediated analysis, not for Node, Bun, or Python. It attempts to derail scanners or analyst copilots that feed the beginning of a file to a language model without clearly isolating the content as untrusted data. In weak pipelines, this can cause refusal behavior, prompt confusion, context pollution, or premature classification before the scanner reaches the actual malware...

Step 3.7 Flash shipped on May 29, 2026 as a structural upgrade to 3.5 Flash: same OpenAI-compatible SDK, new vision encoder, new runtime escalation, and a compute-control flag you can set per request. The migration from 3.5 is two environment variables. One of them has to be exactly right — or every call returns a silent 401. What 3.7 brings that 3.5 didn't Step 3.7 Flash adds three net-new capabilities over 3.5 Flash: a native 1.8B-parameter ViT encoder that injects image representations directly into the language backbone without a separate model call , an automatic Advisor Mode that routes failure-prone subtasks to a larger model at runtime, and a reasoning_effort parameter (low / medium / high) as a first-class API flag rather than a prompt-engineering convention. The production-relevance number is variance: 3.5 Flash scores ranged from 43% to 73% across different harnesses ; 3.7 narrows that to 64.5–71.5% , which matters more for production scheduling than the raw score improvement. Quick Answer: Step 3.7 Flash is an OpenAI-SDK-compatible model — model string step-3.7-flash , base URL https://api.stepfun.ai/v1 (global) or https://api.stepfun.com/v1 (China region). New over 3.5: native vision input, automatic Advisor Mode escalation, and a reasoning_effort flag. The only breaking change from 3.5: base URL must match your account region exactly, or you get a 401 with no error body. The architecture is a 198B sparse MoE model with roughly 11B parameters active per forward pass — dense-10B compute cost at much larger capacity. SWE-Bench Pro improved to 56.3% from 51.3% ; Terminal-Bench 2.1 improved to 59.5% from 53.4% , suggesting the planning and shell-operation gains that matter for coding agents are consistent across benchmarks. Advisor Mode carries the headline cost claim from StepFun's internal harness: 97% of Claude Opus 4.6's coding performance at $0.19 vs. $1.76 per task . That's a vendor figure on a first-party SWE-Bench Verified run — treat it as directio

What flipped in b9437 Build b9437 , published on May 30, 2026 at 20:56 UTC , ships two targeted default-value corrections to llama-bench . Flash attention ( -fa ) shifts from a hard-coded off to auto ( LLAMA_FLASH_ATTN_TYPE_AUTO ), and the GPU-layer count ( -ngl ) changes from the legacy sentinel 99 to -1 . Both values now match what llama-server and llama-cli already used — the bench tool was simply never updated to track them until this build. Quick Answer: Before b9437 (published May 30, 2026) , llama-bench hard-coded -fa off , silently skipping flash attention even on CUDA, Metal, and Vulkan hardware. Build b9437 sets the default to -fa auto and -ngl -1 , matching llama-server and llama-cli . Any pre-b9437 baseline on FA-capable hardware needs a flag-matched re-run to remain valid. PR #23714 , reviewed and merged by maintainers JohannesGaessler and pwilkin, adds the same -fa auto|off|on tri-state flag to llama-bench that the rest of the toolchain already supported. With LLAMA_FLASH_ATTN_TYPE_AUTO as the new default, flash attention activates automatically when the runtime detects a capable backend (CUDA, Metal, Vulkan); on CPU-only hosts it stays off with no error and no output change. Parameter Before b9437 After b9437 Behavioral impact -fa off (hard-coded) auto ( LLAMA_FLASH_ATTN_TYPE_AUTO ) GPU-capable hosts bench with FA active by default; pre/post comparisons require explicit flag-matching -ngl 99 (offload-all sentinel) -1 (runtime decides) CPU-only builds no longer attempt full GPU offload; eliminates spurious CUDA errors when no GPU is present The following verified script (executed successfully, exit 0) demonstrates the behavioral gap in concrete terms — on a capable GPU, the pre-b9437 defaults schedule zero FA rows while b9437 defaults schedule one: def old_llama_bench ( device ): # Before b9437, the default bench matrix used FA=0, so FA rows were skipped. return [{ " device " : device [ " name " ], " ngl " : 0 , " fa " : 0 }] def b9437_llama_bench ( de

A recall reference built from working through a 7-layer prompt-injection challenge. Focus: how each defense layer works, where it breaks, and most importantly how to defend. The one idea underneath everything LLMs have no hard boundary between instructions and data . Everything in the context window — system prompt, user message, retrieved documents — is one stream of tokens the model interprets. Prompt injection exploits exactly this: attacker-controlled data gets read as instructions . You cannot fully filter your way out of it; you manage it with defense-in-depth , knowing each individual layer is bypassable. The defense layers (and where each cracks) A progression of controls from weakest to strongest, each with the lesson it teaches. 1–2. No / weak guardrails Baseline: the model just answers. Lesson: an LLM holding secrets in its context with no controls will leak them on request. 3. Input filtering — block words in the user's message Defense: scan the incoming prompt for banned terms ("code", "secret", "reveal") and block. Weakness: keyword blocklists are trivially evaded — synonyms, misspellings, split words, leetspeak, another language, oblique references. Filtering strings doesn't filter intent . What actually helps: prefer allowlists to blocklists; classify intent semantically rather than matching keywords; treat all input as untrusted; rate-limit and log probing. 4. Output filtering — catch the secret in the response Defense: string-match the known secret in the model's output and redact. Weakness: substring matching only catches the contiguous secret. Fragmenting or transforming it (separators, per-character, encodings) means the literal string never appears, so there is nothing to match. What actually helps: don't put secrets where the model can emit them in the first place; minimize sensitive data in context; treat output filtering as a brittle last line, never a primary control. 5. Input + output filtering combined Defense: both of the above, stacked.

There is a dangerous trap in the local AI world: picking the smallest quantization that fits into your VRAM just because it "runs." We see developers doing this all the time, completely unaware that they’ve crippled their agent's ability to reason. It’s easy to look at a leaderboard, see a model rank high, and assume it’s good to go. But leaderboard scores are a poor proxy for real-world agent behavior. A model might pass a static benchmark at a lower quantization, but when you put it in an agentic loop, its tool-calling accuracy can fall off a cliff. We built the "Quant Audit" feature in QuantaMind because we were tired of this silent failure. It systematically measures the performance drop-off as you move through different compression levels. The goal shouldn’t be to find the smallest quant that loads; it should be to identify the largest quant that actually retains the reasoning integrity your app requires. Stop guessing, start measuring, and stop letting leaderboard hype dictate your architecture.

The suggestion every RAG app ignores If you've shipped a retrieval-augmented assistant, you've written some version of this line in your system prompt: "If the answer isn't in the provided context, say you don't know. Do not make things up." And you've watched the model cheerfully ignore it under pressure. A confident-sounding question comes in, retrieval returns something adjacent , and the model stitches together an answer that's plausible, fluent, and wrong. Telling a language model not to hallucinate is a suggestion — and suggestions lose to the model's overwhelming prior toward being helpful. I got tired of fighting this with prompt wording, so I tried a different framing while building MCP SDK Docs Assistant , an assistant for the Model Context Protocol TypeScript SDK. The framing: don't ask the model to refuse — remove its ability to fabricate. Refusal as code, not as a prompt The core idea is that the model can only hallucinate if you hand it material to hallucinate from. So the refusal decision lives in the retrieval tool, before the model ever sees anything. If nothing clears a confidence bar, the tool returns an empty result set, and the model is left with no source text to spin into an answer. In practice, the tool looks roughly like this: const candidates = await hybridSearch ( query , { version , limit : 12 }); if ( ! hasConfidentMatch ( candidates )) { // best cosine sim < 0.45 return { relevant : false , results : [] }; // model has nothing to work with } const results = await rerank ( query , candidates , 6 ); return { relevant : true , results }; The model isn't asked to behave. The system is shaped so that the only coherent next move, when results come back empty, is to say "the docs don't cover this." Refusal stops being a personality trait you're hoping for and becomes a property of the architecture. Why this particular SDK needed it There's a second failure mode this assistant had to solve, and it's specific to fast-moving libraries. The MCP Ty

"How does ChatGPT think ?" It doesn't. The entire mechanism behind every chatbot is almost anticlimactic: it predicts one next word , adds it, and repeats. I built a tiny interactive predictor so you can be the model — and it explains both the magic and the flaws. 🔮 Be the model: https://dev48v.infy.uk/ai/days/day6-next-token.html This is Day 6 of AIFromZero — AI literacy, one concept a day, no code to follow. 1. It only predicts the NEXT word Given everything so far, the model outputs a probability for every possible next word, picks one, appends it, and runs again with the longer text. Paragraphs, code, poems — all of it is this one step on repeat. "the cat sat on the ___" → P(mat) high, P(bird) low 2. It's a probability over the WHOLE vocabulary The output isn't one word — it's a number for every word it knows (100,000+ for a real model). Most are near zero; a handful are plausible. The bars in the demo are that distribution, over a tiny vocabulary. 3. Autoregression: feed the output back in After picking a word, it becomes part of the input for the next prediction. Predict → append → predict again. Because each new word conditions on all the previous ones, short local choices add up to coherent long text. 4. Temperature = the creativity dial Once you have probabilities, how do you choose? Temperature reshapes them before sampling: Near 0: the top word always wins — safe, repetitive. High: the odds flatten, so rarer words get a real chance — creative, error-prone. p = p ** ( 1 / temperature ); // then renormalise and sample Drag the slider in the demo and watch the bars sharpen or even out. That one knob is what an API calls "creativity." 5. Where do the probabilities come from? In my toy, from counting which word followed which in a few sentences (a "bigram" with 1-word memory). A real LLM replaces the counting with a giant neural network trained on much of the internet, and its memory spans thousands of words. The mechanism is identical — only the quality of th

Lars Faye's Agentic Coding Is a Trap — published Sunday, May 3, picked up on Hacker News at 398 points and 316 comments — is the best single compendium of the cognitive-debt evidence base anyone has put together in 2026. It catalogues the studies. It names the trade-offs. It lands on a personal-discipline conclusion. The receipts are now collected; the careful reader will have spent the weekend nodding through them. Buried in Faye's second paragraph, almost in passing, is the line that does the actual analytical work. Faye describes the agentic workflow as a process in which "someone defines the project's requirements ... generates a plan, and then pulls the slot machine lever over and over, iterating and reiterating with often multiple agent instances until it's done." The link goes to a March post by Quentin Rousseau, CTO and co-founder of Rootly, titled One More Prompt: The Dopamine Trap of Agentic Coding. The metaphor isn't Faye's. Rousseau got there first, in clinical language: the workflow runs on "variable ratio reinforcement — the same psychological mechanism that makes slot machines the most addictive form of gambling" . That is the framing the rest of Faye's piece is downstream of, and it is the framing this article is about. What the receipts add up to Faye's catalogue, briefly. Anthropic's own research note on internal use names what it calls the "paradox of supervision" : effective use of Claude requires the very skills that sustained Claude use atrophies. MIT Media Lab's Your Brain on ChatGPT measured the cognitive impact and labelled it cognitive debt . A Microsoft study covered by 404 Media reached parallel findings for knowledge workers more broadly. A separate Anthropic study on coding skills reported a 47% drop-off in debugging skills among engineers leaning heavily on AI-assisted workflows. Sandor Nyako, the LinkedIn engineering director who oversees fifty engineers, has reportedly asked his team not to use these tools for "tasks that require cri

On 14 April, the Trump administration quietly acknowledged the widespread use of AI to automate government processes. The office of management and budget (OMB) disclosed a staggering 3,611 active or planned use cases for AI across the federal government. The list has ballooned by 70% from the one published in the final year of the Biden administration, and includes many disturbing-seeming plans to hand over sensitive governmental functions to AI. Scanning this list, many readers may find many causes for alarm. It represents a transfer of decision processes from human to machine on a massive scale over matters of individual freedom, public health and well-being, nuclear reactor safety and more...

json.loads(response) fails at a certain point. You told the model "return JSON only," but it added a ```json markdown code fence around everything. A quick regex strips it — until that regex hits an edge case, and that edge case blows up in production. Since Ollama 0.3.0, passing a JSON schema to the format parameter eliminates this problem at the root. The model's inference itself is constrained by the schema, so no code fences, no explanatory text, no mid-thought artifacts. Just parseable JSON. I ran these tests locally with Gemma4 and Ollama 0.30.7 to see how well it holds up in practice. Why LLM Response Parsing Is Tricky The most common problem when running Ollama locally — without a cloud LLM API — is JSON parsing. Two reasons. First, text generation models are trained toward "natural text." Even if you ask for JSON only, they'll often wrap it in json ... blocks or prepend "Of course! Here is the JSON you requested:" style text. Here's what I reproduced directly: json Input: 'Give me 3 Python tips as JSON with keys: tips (array), difficulty (1-5)' Model output (no format parameter): ```json { "tips": [ "Master the fundamentals first...", ... ] } JSON parse: FAILED Python ' s `json.loads()` can ' t handle the markdown wrapper . The " JSON only " instruction is unreliable in production . Second , speed . I measured the same query both ways : 32 seconds without structured output , 5 seconds with it . More on why below . ## How the Ollama format Parameter Works Ollama ' s `/api/generate` endpoint has a `format` field. Pass a JSON schema object and Ollama applies **constrained decoding** during inference. python import json import urllib.request def ollama_structured(prompt, schema, model="gemma4:e4b"): payload = { "model": model, "prompt": prompt, "format": schema, # ← pass JSON schema object directly "stream": False, "options": {"temperature": 0} } data = json.dumps(payload).encode() req = urllib.request.Request( " http://localhost:11434/api/generate ", data=data

Coding agents draft specs, architecture docs, changelogs, and README updates in seconds — but a human still has to judge the quality of all that output. The bottleneck shift A year ago, the typical workflow was: you write a spec, you get comments, you revise, then you implement and get code review. Humans did most of the writing and coding. Now, agents produce first drafts of design docs, API references, runbooks, and onboarding guides — and they do it in seconds. Code implementation and code review can now be handled by agents, so those are no longer the bottleneck. What surfaced instead is the step right before: document review. A human has to read 2,000 lines of generated markdown and decide what's wrong. The writing part got dramatically faster. LLMs can assist with document review too, but compared to code implementation and code review, the human judgment required is still larger. This asymmetry compounds fast. Every agent-assisted project now has a stack of "needs human review" documents growing in a shared folder. If you're running multiple agent loops in parallel — one for the spec, one for the implementation plan, one for the test strategy — review becomes a pipeline stall. GitHub PRs remain the right tool when you need third-party review. But the step before that — the fast local self-review loop where you and your agent iterate on a draft — doesn't belong in a PR. Branching, diffing, and assigning reviewers is a lot of process for a first draft the agent wrote in seconds. Why prose feedback is lossy The most common workaround today is to have the agent read the document and then fix things based on natural-language feedback: "The error handling in section 3.2 is too vague — be specific about what happens on timeout." This looks reasonable. The agent reads it, searches for something about error handling, and makes a change. But several things go wrong: Position is ambiguous. If section 3.2 has three paragraphs about error handling, which one did the revie