标签：#LLM

I built a pytest harness that runs the same set of questions through five language models at once - a free local Llama, plus GPT, DeepSeek, and two Claude models - and compares them on the three things a team pays for: cost per query, speed, and answer quality. The plan was simple. Run the grid, read the scoreboard, say which model to use. The scoreboard came back clean and easy to read. This is the story of why I didn't trust it, and what I found when I checked. The thing I stopped trusting wasn't any of the models. It was the tool I was using to score them. It's also the first project in this series that spends real money. Every one before it ran locally, for free. Here each call costs something, and the whole comparison came to about 21 cents. That price is small, but it changed how I tested, and not in the way I expected. The scoreboard, and why I didn't stop there Five models, the same ten questions, twice each, every call measured. Here is the run, ordered by quality score (a second model grades each answer on correctness and relevance, combined into a 0-1 score, pass line 0.7): model quality mean $/query mean latency out-tokens deepseek-v4-pro 0.970 $0.000138 2713 ms 113 claude-haiku-4-5 0.967 $0.000537 1597 ms 104 gpt-5.6-luna 0.962 $0.000082 1323 ms 65 claude-sonnet-5 0.937 $0.002426 4093 ms 239 llama3.2 (local) 0.922 $0.000000 7859 ms 130 Read it straight and it looks finished. The whole quality column sits in a tiny band, 0.92 to 0.97. The cheapest, fastest paid model scores right in there with the rest. The most expensive one, Sonnet, at about thirty times the price per query, sits no higher than the others - its answers are just longer (239 tokens to GPT's 65), which costs more and takes longer without scoring better. So the easy takeaway is: use the small cheap model, skip the expensive one. I want to be careful with that, because it's the kind of tidy result I've learned to distrust. The gaps between the top models are tiny, and a ranking built on tin

The Forked History: Byzantine Witness and the 3-of-4 Quorum — Tested Agent Determinism Illusions (Part 19) 2026-08-20 Where this fits: Part 18 closed the runtime face of C3's boundary at capability isolation — the oracle reads from a surface the producer cannot write. Part 18's §6 named the residual this part answers: Byzantine authority . Sealing one honest oracle's history says nothing about whose view is the truth when a compromised authority can present forked views to different observers. This part maps the witness layer — the answer is not a stronger single authority, but a witness set with three separable properties, an explicit fault bound, and a governed membership surface. Part 18 ended with the oracle isolated from the producer's writable surface. Isolation answers "can the producer fake the read?" It does not answer "whose read is the truth when the authority itself equivocates?" A trusted parent that closes the verdict channel still presents the approval history. If that authority is compromised — or the CI choosing the harness is — it can show job A a signed checkpoint and job B a different fork; each job sees a locally valid tree head with an inclusion proof, and no single history exists. This part tests that shape, then the witness machinery that answers it. 1. Sealed floor ≠ global history Peter's pin-rollback reply sealed a monotonic minimum-version floor: CI could no longer resurrect an older harness with a known false-green channel. The sealed floor is honest for what it claims. It is not what it looks like at first. The split into two predicates. The sealed floor proves this job did not go backwards on the view it was shown. It does not prove the approval history itself is one global append-only log. A compromised authority can hand job A a signed checkpoint whose minimum is 2 and job B a fork whose minimum is still 1; each view carries a locally valid signature and an inclusion proof while no single history exists. cell setup result A local sea

Part 4 of the Building the AI Memory Stack series After finishing the previous article, I looked at...

TL;DR: We built a one-shot HTTP worker that talks to Notion through MCP. Version one worked. Version two got cheaper and more readable, then failed in a new way. The harness was fine. The tool surface, the model, and the prompt were not the same problem, and we kept treating them as one. We keep seeing the same pitch: put an agent in the cloud, give it tools, let it live in Slack, let it remember you. That is a product. It is not the product we needed. We needed something dumber and more useful. Another service should be able to say "read this Notion page, write a summary somewhere, stop." No chat history. No personality that accretes over weeks. No always-on process. If nobody is calling it, it should cost nothing. We started calling that shape a one-shot agent . One HTTP request. Tools for that request. A JSON result. Then the instance can go away. This is the path we actually walked: first working version, what it got wrong, the Markdown fork, and the cheaper tricks that mattered more than swapping frameworks. The job was never "build a chatbot" The first real task was almost boring. Once a week, pull a skill write-up from Notion, extract what mattered, and append it to a digest page. Callers would name pages in English. They would not paste Notion ids. If a name was ambiguous, the agent should refuse to write rather than guess. If that loop is wrong, people stop trusting write-back. If it is expensive, nobody schedules it. If it needs a human to babysit a terminal, it is not a system. So the constraints were social as much as technical: An external caller owns the schedule. The agent does not. The agent must be allowed to use tools, not just talk about them. Secrets stay in the environment, never in the request body. Idle time should be free. Question you will probably ask: why not a cron script that hits the Notion API directly? Because the task changes every call. This week it is a weekly digest. Next week it is "list in-progress rows and do not write." We did

Prompt engineering is a conversation; context engineering is system architecture. In the early days of working with Large Language Models (LLMs), optimizing the prompt was enough for simple text generation tasks. But when you are building autonomous systems—like a self-hosted automation server connecting cloud databases, webhooks, and reasoning nodes—prompts alone will not keep track of APIs, past decisions, and strict output constraints. Think of the LLM as the CPU, and the context window as the RAM. Context engineering is the discipline of treating that memory as a scarce resource, meticulously designing the pipeline that feeds the model the exact facts, instructions, and tools it needs at the precise moment it needs them. The Four Core Strategies To shift from vibe-coding a chatbot to architecting a resilient multi-agent system, you must manage what enters and stays in the context window using four primary techniques: Select: Decide exactly which external sources—like database schemas or specific API documentation—enter the context window to maximize the signal-to-noise ratio. Compress: Shrink the context payload only after the key facts are successfully structured. Write: Persist the task state and intermediate decisions outside the active context window so the agent can retrieve them later. Think of this as giving the agent its own local-first markdown vault for networked thought. Isolate: Separate contexts when domains collide. Instead of forcing one model to do everything, build multi-agent systems where each agent receives a strictly scoped slice of the context. Navigating the Failure Modes Stuffing a massive context window with raw JSON logs and unstructured data is a recipe for disaster. When building complex workflows, you must engineer guardrails against these critical failure modes: Context Poisoning: Hallucinated or incorrect information enters the context and compounds over time because the agent continually reuses it. Context Distraction: The agent g

I gave my LLM a 29-question order-reading exam. Last time was how to build the exam. Today: grading. Grading gets its own post for a reason. Build the grading wrong, and the score lies to you. 5 wrong out of 29 — can I ship? No idea. Because "which 5" is missing. If it missed 5 typo-riddled questions, ship it. But if one of those 5 was reading "please cancel my order" as a NEW order? Then even with everything else perfect, you can't ship. That program sends goods to a customer who just cancelled. So don't grade by count. Grade by severity. Severity = "can a human undo this?" My grader has 4 grades. One criterion — is it reversible? In this program, the irreversible moment is when the wrong goods get loaded onto a truck. FATAL Wrong goods on the truck. Cannot be undone RISKY Confirmed something ambiguous without asking. Right this time — fatal next time MISSED Dropped an order. The customer calls. Fixable HARMLESS Over-asked "please confirm." Just slower One principle falls out of this: A wrong confirmation is worse than no confirmation. Sounds obvious. In production you'll be tempted to flip it. Someone complains "it asks for confirmation too often," so you lower the confidence bar. The screen gets cleaner. And the accidents start happening off-screen. The same 28/29 splits two ways FATAL 0 · MISSED 1 → Ship it. Humans catch what it drops FATAL 1 · everything else perfect → Don't ship. You don't know when that 1 comes back Same score. Opposite fates. Two accidents my grader caused The grader is code I wrote. Like all code I write, it had bugs. Accident one — zero points over formatting. A model answer was perfect in content, but the JSON wrapper arrived with the tail cut off. The grader ruled "broken format = fatal." A 100-point answer, zeroed over one missing brace. The fix is simple: count the open brackets and close what's missing (ignoring brackets inside strings). The actual code is in parse_json in the repo . Accident two — penalizing a good answer. For "250 b

Last time I gave my LLM an order-reading exam and lost 5 times as the exam author. Today: how that exam was built. Conclusion first — nice questions are a waste of paper. You'll want to start with the happy path Ask anyone to write a test and they start with the case that works. "5 boxes of the 250 shipping boxes please" → shipping box 250, 5 boxes. It passes. Feels good. Reassuring. But that's wasted points. Models rarely fail the normal cases. What fails is everything that isn't normal. My 29 questions broke down like this: Normal orders 4 Things that aren't orders 6 ← the biggest group Changes & cancellations 4 Ambiguous ones 5 Typos & extreme shorthand 3 After learning kicks in 7 Normal is the smallest group. On purpose. Why "not an order" gets the most questions The worst accident for this program is shipping something nobody ordered. So the exam should aim at that accident more than anything else. What are the dimensions of the 250 shipping box? Product name: present. Number: present. But it's not an order. It's a question. A program that treats "product name spotted" as "order detected" calls the truck right here. So I planted six of these: price inquiries, stock inquiries, delivery questions, greetings, a tax-invoice request. Changes and cancellations are nastier. I ordered 5 boxes of the 250 — please send only 3 Two numbers. Read only the first half and it's a perfect order. Treat it as a new order and the goods ship twice. Plant traps in the catalog too It's not just about hard questions. Make the data itself messy. Two kinds of clear tape — 48mm and 60mm Five products starting with "250" Different pack sizes per box — 50, 40, 25, 10 sheets A few loose items with no box unit at all One reason: real data already looks like this. A real product catalog always has near-twins. Run the exam on a clean catalog and here's what happens — everything passes. Then you plug in production data and it collapses. If the exam passed but production has accidents, that's no

🛡️ Arquitectura de Defensa para Agentes de IA: Cómo asegurar tus LLMs contra Prompt Injection, Tool-Poisoning y Fugitividad. El ecosistema actual de agentes autónomos y servidores MCP (Model Context Protocol) es brillante, pero operativamente es una pesadilla de seguridad. Estamos construyendo sistemas que ejecutan código, acceden a bases de datos y toman decisiones críticas basándose en salidas de texto que son intrínsecamente manipulables. Si estás desplegando agentes en producción confiando únicamente en que el modelo "se portará bien" gracias a su System Prompt, estás completamente desprotegido. Para solucionar esto, he desarrollado un framework de defensa en profundidad distribuido en 4 capas críticas. No se trata de teoría académica; son sensores deterministas y dinámicos listos para producción. 🏗️ Las 4 Capas del Ecosistema de Seguridad. Capa 1: Sanitización de Entrada y Control de Estímulos (Ingress). El ataque empieza antes de que el modelo procese el token. Necesitamos interceptar vectores maliciosos tanto en texto como en medios visuales. hermes-shield: Un sanitizador de entrada anti prompt-injection que opera en 5 capas secuenciales para neutralizar instrucciones hostiles ocultas. vision-injection-guard: Un sensor determinista diseñado para procesadores VLM. Detecta texto malicioso inyectado visualmente en imágenes antes de que el modelo de lenguaje lo interprete. corpus-scrub: Herramienta de prevención de fuga de datos. Detecta y redacta de forma automática PII (información personal identificable) y secretos en tus corpus de datos antes de entrenar o ajustar un LLM. Capa 2: Pasarela de Control y Validación Física (Gateway & Sandbox). Una vez que el agente está activo, no puede comunicarse directamente con el exterior sin un proxy que valide sus intenciones. ai-guard-gateway: Una pasarela de seguridad profesional para endpoints expuestos. Implementa Rate Limiting, redacción de PII en tiempo real, detección de inyecciones y políticas OPA (Open Policy Agen

There is a small cluster of posts going around right now about auditing your LLM invoice, and about how cost calculators get the numbers wrong. I went to check mine and hit a problem before I got to the arithmetic: my pipeline doesn't produce an invoice, and the plumbing I built two months ago is the reason why. This project has a script, git_commit.py , that turns a staged git diff into a Conventional Commit message. It shells out to the Claude CLI. There is no ANTHROPIC_API_KEY anywhere in the project, on purpose — an early version used urllib against the API directly and broke immediately for anyone running on an OAuth session instead of a raw key, so every AI call in the repo goes through a claude -p subprocess instead. That decision is still right. It also means there is no API key, so there is no per-key usage dashboard, so there is no line item to audit. For several months this script has been making a model call on essentially every commit, and I have never once known what any of them cost. The call site throws the numbers away Here is the actual invocation, trimmed: raw = subprocess . check_output ( [ " claude " , " -p " , " --safe-mode " , SYSTEM + " \n\n " + diff ], text = True , timeout = 20 , env = _claude_subprocess_env (), ) subprocess.check_output returns stdout. With the CLI's default output format, stdout is the commit message string and nothing else. Every number I would want — tokens in, tokens out, dollars — is computed on the other side of that call and then discarded, because I asked for a string and a string is what I got. This is the part I want to flag for anyone wiring up a headless model call the same way. It isn't that the metering is missing. It's that the default output format is lossy in exactly the dimension you'd later want to audit, and you won't discover that by reading your own code, because your own code looks fine. It asks for text, it gets text. The fix is one flag: raw = subprocess . check_output ( [ " claude " , " -p " , " -

I spend a lot of time in the AI space -- reading papers, building things, talking to engineers who are actually shipping. And there is a gap between what the demos show and what production systems actually look like that nobody is being fully honest about. So here is my honest take on where things actually are. The Problem With How We Talk About AI Agents Everyone is calling everything an "agent" right now. A function that calls a tool? Agent. A chatbot with memory? Agent. A script with a loop? Agent. This dilution is not just semantic. It is causing real engineering mistakes. When you do not have a precise definition for what you are building, you end up over-engineering simple pipelines and under-engineering genuinely complex ones. I have seen teams spend weeks adding "agentic" orchestration to workflows that would have been fine as a single well-structured prompt. Here is the definition I keep coming back to: an agent is a system that has an objective, not just an instruction. It decides what to do next. It handles failure. It knows when it is done. Everything else is just a fancy function call. 🟢 If your system needs a human to tell it each step, it is not an agent. It is a chat interface. 🔵 If your system can recover from a failed tool call and try a different approach, you are getting somewhere. ✅ If your system can decompose a goal into subtasks and delegate them, that is the real thing. What Is Actually Happening in Production Right Now The honest picture from teams I follow and talk to: Most real agent deployments are narrow. They do one thing well. Customer support triage. Document extraction. Code review on a specific codebase. They are not general-purpose reasoning engines. They are purpose-built pipelines with some intelligence in the decision layer. The teams getting good results are not chasing the latest model release. They are obsessing over: ☑️ Tool design -- what can the agent actually call, and how clean is the interface ☑️ Failure handling -- wh

Choosing the Right GPU for Your Model — A Sizing Method, Not a Guess OK, you're a senior SRE, you've been hearing incessantly about AI models, but aren't quite sure how to determine the correct node size to host your model. - If so ... you're in the right place. Part of a series on running vLLM on AKS. Companion piece: How to avoid flapping . GPU infrastructure setup — coming soon. This piece walks through estimating GPU memory requirements from both a model's parameter count or a concurrent requests requirement. After reading this article you will have enough knowledge to pick a GPU family with confidence. Disclaimer: this process is a rule-of-thumb filter, not a precise calculation — the last step covers how to get exact numbers once the model is actually running. Background: What actually consumes GPU memory AI models live in GPU memory — VRAM — and engines such as vLLM provide novel techniques for managing that memory efficiently [ paper ], but the model isn't the only thing consuming it. Below is a short list of things that consume our precious VRAM: Model weights — the parameters themselves. The big fixed cost: loaded once, never shrinks. KV cache — working memory for in-flight requests. Every token of every active request holds its attention keys/values here. This is the one that determines throughput : more KV cache = more concurrent requests. Everything else — activations (the temporary tensors of a forward pass) plus CUDA/framework overhead. You don't calculate these by hand; vLLM measures activations with a profiling pass at start-up and prints it for our consumption. The sizing question is really: after weights and overhead, how much is left for the KV cache — and is that enough for your traffic? OK, lets get started Step 1 — Choose a model Guidance on which model to choose is outside the bounds of this article. What matters here: once you have a candidate, everything below can be read off its spec sheet — you can then run this method on every model on y

AI Agents Don’t Need More Context. They Need Memory. S6stem S6stem S6stem Follow Aug 17 AI Agents Don’t Need More Context. They Need Memory. # ai # programming # agents # rust 2 reactions 2 comments 4 min read

Short answer: For cheap bulk CSV tagging, use an asynchronous LLM text classification API, estimate each tenant batch before it runs, and attach the eventual export to the same tenant ledger instead of sending one request per row. For a one-person B2B SaaS, the useful comparison is not a model leaderboard. It is the amount of accounting and integration work left in the product after classification finishes. Option Choose it when Tenant-cost consequence Catch Infrai You want a self-describing REST API whose public discovery supplies request and response schemas plus runnable examples One key and one bill make the external side of reconciliation smaller Moderation uses chat classification with JSON Schema because there is no dedicated moderation endpoint OpenAI direct Your product has already standardized on OpenAI Keep tenant attribution in your own job ledger A direct contract does not remove application-level CSV reconciliation Anthropic direct Your model decision is already Anthropic-specific Use the same internal ledger pattern You own the provider-specific adapter and export mapping Google Gemini direct Your model decision is already Gemini-specific Use the same internal ledger pattern You own the provider-specific adapter and export mapping Recommendation: use asynchronous chat classification with a closed label set, but treat the tenant ledger as the primary artifact and the provider batch as an execution detail. That keeps a nightly backfill away from the request path and makes every charge explainable before a human moderator sees the result. The model matters. The accounting boundary matters more. Start with the allocation unit, not the provider A moderation upload arrives as a CSV, but a CSV is a transport format, not a billing unit. The billing unit should be an immutable application job owned by one tenant. Give that job an internal ID, record the source-file identity, preserve the row identifiers, and bind the approved label vocabulary to it. Then estim

How a theory of leftover surprise changed a memory layer Richard Emate Richard Emate Richard Emate Follow Aug 18 How a theory of leftover surprise changed a memory layer # python # ai # llm # opensource Add Comment 9 min read

OpenAI has cut prices on its frontier models again. If you're running any production workload on the API, your cost assumptions from six months ago are probably stale. The Real Impact of a Pricing Halving A 50% price cut sounds like pure good news, but it changes the calculus on decisions you already made. Projects you shelved because the token costs didn't pencil out deserve a second look. Architectures you built around cheaper, less capable models to save money may now be false economies - the cost gap between "good enough" and "best available" just got smaller. The more interesting shift is for teams running retrieval-augmented generation (RAG) pipelines - systems that pull relevant documents from a database at query time and feed them into the model as context. RAG workflows tend to be token-heavy because every retrieved chunk counts against your input token bill. At the old pricing, teams were aggressively trimming context windows and limiting retrieved chunks to stay within budget. At half the cost, you can retrieve more, keep longer context, and let the model reason over richer information - without changing a line of retrieval logic. Real Example Here's a simplified cost check you can drop into any project that calls the OpenAI API: import openai # Approximate pricing per 1M tokens (check platform.openai.com for current rates) INPUT_COST_PER_1M = 2.50 # update to current figure OUTPUT_COST_PER_1M = 10.00 # update to current figure def estimate_cost ( input_tokens : int , output_tokens : int ) -> float : return ( input_tokens / 1_000_000 * INPUT_COST_PER_1M + output_tokens / 1_000_000 * OUTPUT_COST_PER_1M ) # Example: a RAG call with 3,000 input tokens and 500 output tokens print ( f " Estimated cost per call: $ { estimate_cost ( 3000 , 500 ) : . 5 f } " ) # Run this across your monthly volume to see the real delta Multiply that per-call number by your actual monthly call volume and compare it against what you budgeted. For many teams, the difference will jus

I have been thinking a lot about AI and integrity. Part of that is contextual integrity. I recently found two papers on the topic. “ CIMemories: A Compositional Benchmark for Contextual Integrity of Persistent Memory in LLMs “: Abstract: Large Language Models (LLMs) increasingly use persistent memory from past interactions to enhance personalization and task performance. However, this memory introduces critical risks when sensitive information is revealed in inappropriate contexts. We present CIMemories, a benchmark for evaluating whether LLMs appropriately control information flow from memory based on task context. CIMemories uses synthetic user profiles with over 100 attributes per user, paired with diverse task contexts in which each attribute may be essential for some tasks but inappropriate for others. Our evaluation reveals that frontier models exhibit up to 69% attribute-level violations (leaking information inappropriately), with lower violation rates often coming at the cost of task utility. Violations accumulate across both tasks and runs: as usage increases from 1 to 40 tasks, GPT-5’s violations rise from 0.1% to 9.6%, reaching 25.1% when the same prompt is executed 5 times, revealing arbitrary and unstable behavior in which models leak different attributes for identical prompts. Privacy-conscious prompting does not solve this—models overgeneralize, sharing everything or nothing rather than making nuanced, context-dependent decisions. These findings reveal fundamental limitations that require contextually aware reasoning capabilities, not just better prompting or scaling...

You typed it. Claude rewrote it. Then it watermarked its version and shipped it to your reader without telling either of you. Last week, Anthropic's text adulteration watermarking became the most-discussed AI topic on Hacker News. Daring Fireball called it a perversion of writing. The thread hit 762 points and 673 comments. The same week, Anthropic reported 65 billion dollars in annualized revenue, and a separate debate over Claude's war on open-source AI added 133 points to the conversation. I write technical articles on Medium, Dev.to, and LinkedIn. I use Claude as an editing assistant. When I read the watermarking coverage, I realized this is not a technical debate about whether watermarking works. It is a fight over who owns the words you write with AI help. Here is what is actually happening, why writers and developers are angry, and what it means for anyone who publishes online. What the watermark actually does Anthropic's documentation (watermarking and attribution) explains the feature this way: text adulteration watermarking adds invisible signals to Claude's output so that services can detect whether text was generated by AI. The watermark survives copy-paste, paraphrasing, and light editing. If a platform integrates Anthropic's detection tool, it can flag AI-generated content even after the text has been modified. The controversy is not that watermarking exists. The controversy is that Claude applies this watermark to content the user wrote themselves, if that content passes through Claude's interface. Here is the scenario from the angry commenters: You write an email in a text editor. You paste it into Claude and ask: Clean this up, make it shorter, fix the grammar. Claude rewrites your email, applies the watermark, and returns the text. You copy that text into your email client and hit send. The recipient's email system, if it uses Anthropic's detection, flags your email as AI-generated. You wrote the original words. You directed the rewrite. You approv

Hello, I'm Maneshwar. I'm building git-lrc, a Micro AI code reviewer that runs on every commit. It is...

AI evals and analysis Let's say you're testing out new AI tools. Perhaps you implement and...

If you love the multi-agent AI orchestration concepts from Python’s CrewAI, but want the performance, native concurrency, and low memory footprint of Go, check out crewai-go. The v0.4.0 release brings key capabilities to make building multi-agent systems in Go fast, type-safe, and production-ready. ✨ Key Highlights: 🛠️ Custom Tools: Easily create and bind custom tools using tools.NewTool(...). 🔄 Sequential Context Flow: Outputs from previous tasks flow directly into subsequent tasks as context. 📦 Structured Outputs: Map LLM responses straight into native Go structs using standard json:"..." tags. 🏠 Flexible Provider Support: Run fully offline with Ollama or integrate seamlessly with OpenAI. 🧠 Short-Term Memory: Agents keep context across complex task executions. 💡 Quick Example: package main import ( "context" "fmt" "log" "github.com/rhgs/crewai-go/crew" ) func main () { researcher := crew . NewAgent ( crew . AgentConfig { Role : "AI Researcher" , Goal : "Analyze tech trends" , Backstory : "An expert in discovering high-impact open-source Go tools." , }) task := crew . NewTask ( crew . TaskConfig { Description : "Summarize the main benefits of using Go for AI agent orchestration." , ExpectedOutput : "3 concise bullet points." , Agent : researcher , }) c := crew . NewCrew ( crew . CrewConfig { Agents : [] * crew . Agent { researcher }, Tasks : [] * crew . Task { task }, }) result , err := c . Kickoff ( context . Background ()) if err != nil { log . Fatal ( err ) } fmt . Println ( result . Raw ) } 🔗 Release details & GitHub repo: github.com/rhgs/crewai-go/releases/tag/v0.4.0