标签：#agents

Almost every cost discussion about AI agents opens with a model price per million tokens, which is the one number that tells you the least. The bill you actually receive is a stack of four things: API calls, infrastructure, the one time build, and the recurring costs nobody put in the estimate. Here is how the stack usually breaks down and which layer is worth attacking first. Where The Money Actually Goes For a typical business agent, a support bot or an internal automation running on a managed platform, monthly operating cost lands between 200 and 1,000 dollars. API calls are 40 to 60 percent of that. Hosting, a vector database for memory, and monitoring share the rest. The spread on either side is wide: a solo developer on open source models and a small VPS can stay under 50 dollars a month, while an enterprise running multi agent systems on frontier models regularly spends 5,000 to 13,000 a month before anyone counts the build. Infrastructure has its own shape. Serverless is the cheapest entry, and a moderate agent handling 10,000 to 20,000 interactions a month usually runs 50 to 200 dollars in compute with no idle charge. Containers on ECS, Cloud Run or Kubernetes cost 100 to 500 and buy persistent connections and steady latency. Self hosted GPU starts around 200 a month for a T4 class instance and passes 1,000 for A100 or H100 class, which only pays off at volumes high enough to amortize it. Vector storage adds 20 to 500, and pgvector on a Postgres you already run removes that line entirely. Model Choice Is A Routing Decision The price spread between tiers is large enough that treating model selection as one global choice is the expensive mistake. Frontier reasoning models sit at the top of the range, mid tier models cost a fraction of that, and the lightweight tier is cheaper again by roughly an order of magnitude. An agent that sends every step to the top tier is paying reasoning prices for string formatting. The fix is routing per step rather than per agent

Challenge 2: Dashboard Walkthrough + Nash Demo What I Learned I learned how to navigate the Backboard dashboard and use its built-in features without writing any code. I discovered that the dashboard tracks activity with charts, saves conversations as threads, and supports memory so the assistant can recall facts across different chats. I also learned about Nash, a chat app built on the same platform, which allows me to switch between models seamlessly and even generate images. What I Did I logged into my Backboard account and explored the dashboard. I checked the analytics view to see charts for prompts and tokens. I started a new chat, enabled memory, and told the assistant facts about myself (my name and favorite pizza topping). I opened a new thread and confirmed that the assistant remembered my details. I explored the model library and tried chatting with different models. I visited the memory page to see the facts saved and even deleted one to test the controls. I confirmed my promo credits under Settings. I tried Nash by signing in, starting a conversation, switching models mid-chat, and generating an image. I compared responses from different models to see how they varied. Reflection I realized how powerful memory is for building agents that can recall context across conversations. I also saw how Nash simplifies working with multiple models in one place, which saves time and keeps my data organized. This challenge helped me understand the foundation of agent-based workflows and gave me hands-on experience with multi-modal AI. Screenshot Here’s a screenshot of my dashboard and Nash demo:

NVIDIA Labs open-sourced NOOA (NVIDIA Object-Oriented Agents) this week, and the pitch is unusually simple: an agent is a Python class. Not a graph, not a chain, not a YAML pipeline. A class. I cloned it and got it running the same day. Here's what it actually looks like, what broke, and why I think the core idea matters more than the framework itself. The whole idea in one code block from nooa import Agent class InventoryAgent ( Agent , llm = llm ): """ You are an agent that checks inventory using deterministic helper methods. """ # Plain Python — automatically available as a tool for the LLM def get_stock ( self , item : str ) -> int : """ Get current stock for an item. """ return self . inventory . get ( item , {}). get ( " stock " , 0 ) # `...` body — the LLM implements this at runtime, calling the methods above async def can_fulfill_order ( self , items : list [ str ], budget : float ) -> Result : """ Check if order can be fulfilled within budget. """ ... That's from the repo's quickstart, lightly trimmed. The mapping is: Fields are agent state Methods with real bodies are deterministic tools Methods with ... bodies are implemented by an LLM loop at runtime Docstrings are the prompts Type annotations are contracts the runtime enforces, with auto-retry on mismatch No separate tool-schema JSON. No registration step. The model acts by writing Python in a REPL with access to self , so your method signatures are the tool definitions. Two install gotchas before you try it The README says pip install nooa . Two things I hit on a clean machine: 1. It's not on PyPI yet. As of today, pip install nooa returns No matching distribution found . Install from source instead: git clone https://github.com/NVIDIA-NeMo/labs-OO-Agents.git uv venv --python 3.13 && uv pip install ./labs-OO-Agents 2. No Python 3.14 support. The package pins >=3.12,<3.14 . My default interpreter is 3.14, and the install fails with a version error. Use 3.12 or 3.13. After that, everything imported clean

On August 9, 2026, Ahmad Awais shared a deep dive on X about the read tool in Command Code, his coding agent. The claim is big: the read tool saves billions of tokens a month compared to Claude Code. The full post now lives in the Command Code docs . This article is my summary of that post, written in simple English. If you build agents, or just use them, the lessons are useful. Why a read tool matters Coding agents read files all the time. Every edit starts with a read. Every search result becomes a read. A plan step opens three files. Command Code sees about 50 million reads a month. Each read costs tokens. If one read brings in 500 useless tokens, that is 25 billion useless tokens a month. Worse, those tokens stay in the conversation, and they cost tokens again on every later turn. That is why coding agents feel expensive. The bill is mostly reads, not clever reasoning. Think of the read tool as a compiler. It turns your files into the model's context. Every small choice inside it is a token decision, repeated millions of times. The difference: spend more vs spend less Claude Code's read tool is simple. Ask it to read a 3,000-line file, and it returns all 3,000 lines. Ask for a file with a 3,900-character minified line, and it returns the whole line. No limits at all. That works for Claude Code because its models are strong enough to ignore the noise. It spends more tokens to succeed. Command Code runs on open models. Those models cannot handle a messy read. Users also pay for every token. So Command Code had to spend less. That one constraint forced every design decision below. What Command Code's read tool does differently Three limits, not one. 2,000 lines per file, 128 KB per read, 2,000 characters per line. Each limit stops one kind of bad file: big files, wide files, and minified one-line files. Clear messages instead of silence. If a file is empty, it says "file is empty". If the read goes past the end, it says "try a smaller offset". The model knows what

Sarvar's post about discovering a hidden retry in a 5-agent pipeline (one agent taking 22.6s while others took 5s) is a perfect case study in why observability infrastructure matters for agentic systems. Here's what jumped out: Agent-as-black-box is dangerous. When you string together multiple agents, you lose visibility into retry logic, backoff strategies, and cascade failures unless you instrument at the span level. The latency wasn't in the agent logic itself; it was in the retry envelope. Span hierarchy exposes the invisible. Sentry's approach of grouping spans hierarchically made the problem visible at a glance. Without it, you'd see "agent took 22.6s" and assume it was compute-bound. With hierarchy, the retry pattern was obvious. This scales badly across agents. In a 5-agent system, one bad retry strategy can block or cascade. Add error handling, timeout logic, and fallback chains, and you're building a retry forest no one fully understands. The observability debt compounds. The fix is cheap, the insight is priceless. Once Sarvar knew what was happening, tuning retry counts or backoff curves took minutes. The time cost was finding it. Takeaway: If you're building multi-agent systems, instrument early. Span-level observability isn't optional; it's the difference between "it's slow" and "here's why, and here's the fix."

Cloudflare announced a developer preview that lets any website enable a WebMCP (Web Model Context Protocol) interface with a single dashboard switch. This allows browser-based AI agents to interact with unmodified web pages through structured tools instead of scraping or guessing, keeping human traffic and control on the original site. By Sergio De Simone

A production CMS is a sprawl of endpoints: content types, entries, media, users, webhooks, plugins, settings, admin routes. Hand an agent all of it and the agent gets worse, not better. The model's tool selection drifts as the list grows, and half the tools are things a publishing assistant should never be able to call. The point of this post is the opposite move. Instead of exposing an API and hoping the agent behaves, you curate a small, labeled surface up front. HazelJS Skillgate does that curation from an OpenAPI spec, and that is the part we actually build and run here. Scope, up front This post is about curation and classification: taking a spec with many endpoints and turning a chosen slice of it into governed skills. Skillgate selects the surface, marks read versus write, and would deny destructive methods if they ever entered that surface. Turning a write's approval flag into a real human-approval pause, and enabling an LLM to drive the skills, are runtime concerns handled elsewhere in Agent OS. This demo does not implement them, and this post does not claim it does. What it does show is the curation, and that stands on its own. The tool-explosion problem Point an LLM at a full CMS API and you hit four problems at once: tool selection degrades as options pile up, throughput drops while the model reasons over a long list, you lose visibility into what the agent can actually do, and dangerous operations sit one bad call away. The demo spec here is deliberately smaller than a real CMS, 27 endpoints rather than hundreds, but the problem is identical. Even 27 is too many, and most of them are things a publishing agent has no business touching. From REST endpoint to agent skill Skillgate's input is an ordinary REST API described by an OpenAPI spec: the same entries, media, and user routes a CMS already exposes. Each endpoint is described in the standard OpenAPI shape, a method, a path, parameters, a description, and tags. Two representative operations from the sp

It's 3 AM. A production service is misbehaving, you're on-call, and you'd love an agent that can pull the service's health and tee up a restart for you. The catch is obvious: an agent with raw access to a DevOps API is a liability. One bad call could scale you into a huge bill or delete an incident record you needed. So the real question isn't "can the agent reach the API." It's "which calls should it be allowed to make at all, and how should the dangerous ones be treated differently from the safe ones." That decision is what Skillgate handles, and it's the part we actually build and run in this post. Scope, up front This post is about the classification and curation layer: turning an OpenAPI spec into a governed set of skills. Skillgate decides which endpoints become tools, marks which are read-only, flags which writes should require approval, and denies the destructive ones outright. Wiring an approval flag to a live human-approval pause, and making that pause survive a crash, is the job of the Agent OS runtime, not Skillgate. We link to it at the end. The demo here does not implement that runtime, and this post does not pretend it does. The problem Skillgate solves Point an LLM at a DevOps API and you have three bad options: Expose nothing. The agent is useless. Expose everything. Now the model can call DELETE and scale on a whim. Hand-whitelist every route. It works until the API changes, then it rots. Skillgate replaces all three with opt-in curation plus automatic risk classification. You choose a small surface, and every endpoint on it gets a class based on its method and shape. From REST endpoint to agent skill Skillgate's input is an ordinary REST API described by an OpenAPI spec. Nothing about the API is agent-aware. It's the same deploy, scaling, and incident routes your platform already exposes. Each endpoint is described in the standard OpenAPI shape: a method, a path, some parameters, a description, and tags. A representative operation from the DevOps

Screenshot prompting has a ceiling. You paste the design, the model makes a plausible approximation, you correct it, and on the next turn it drifts again. Nothing is anchored. The model has no source of truth to check itself against between turns. A context bundle changes the contract. Instead of a pixel reference the model has to interpret every time, you get a structured, referenceable set of files — design tokens, layout IR, component inventory, UI strings — that stay in the session and stay consistent. Claude Code can read them, implement from them, and check its own output against them on demand. This post walks the full pipeline, from bundle export to a reviewed, token-verified implementation, using figmascope , a browser tool that turns any Figma file into exactly that bundle. What makes this deterministic Three things make the bundle referenceable rather than interpretable: Tokens are typed and keyed. tokens.json maps semantic names ( spacing.16 , color.7f5cfe ) to exact values. The model can check its output against the file without re-processing the design. The IR is a tree, not pixels. screens/home.json describes the layout in terms of stack/overlay/absolute/leaf nodes — the same abstraction the implementation target (Compose, React, etc.) uses. There's no visual interpretation step. The bundle is stable across turns. Once it's in the repo, every prompt in the session can reference the same files. Token drift is detectable: ask the model to compare its output against tokens.json and it can do it mechanically. Step 1: Generate the bundle Open figmascope.dev in your browser. Paste your Figma file URL. The exporter runs client-side using the Figma REST API — your Figma personal access token is stored in localStorage and never sent to figmascope's servers. Click Export Agent Context . The page exports top-level frames, resolves design tokens, builds the IR, and downloads context-bundle.zip . Step 2: Unzip into your project # from your project root unzip ~/Dow

Long-running agents are usually tested at startup and during normal operation. The awkward middle is ignored: what happens when you need to deploy a new image, rotate a credential, migrate a database, or restart the host while the agent is halfway through a tool call? A process supervisor can restart a crashed agent. It cannot decide whether a browser checkout was committed, whether a webhook was acknowledged, or whether a tool call is safe to replay. That decision belongs in the agent runtime. This post presents a small maintenance-window protocol for agents that run for hours or days. It has four goals: stop accepting new work; let safe work finish or reach a checkpoint; make ambiguous work visible instead of guessing; resume with an explicit recovery decision. 1. Model maintenance as a state transition Do not treat maintenance as kill -TERM followed by hope. Give the runtime a durable state machine: RUNNING -> DRAINING -> QUIESCED -> STOPPED | +-> NEEDS_REVIEW DRAINING rejects new jobs but allows an active job to continue until its next checkpoint or deadline. QUIESCED means there are no unclassified side effects in flight. NEEDS_REVIEW is the safe outcome when the process died after sending a request but before recording the response. Persist the transition, not just an in-memory flag. A minimal record can look like this: { "runtime" : "agent-7" , "maintenance_id" : "mw-2026-08-10-001" , "state" : "DRAINING" , "started_at" : "2026-08-10T08:00:00Z" , "accepting_work" : false , "active_runs" : 2 } If the host disappears, the replacement process can see that the previous shutdown never reached QUIESCED . That is much more useful than inferring health from a missing PID. 2. Put checkpoints around side effects An LLM step is usually replayable. A payment, email, browser click, deployment, or Git push may not be. Record a checkpoint immediately before and after every non-idempotent boundary: PLANNED -> DISPATCHED -> ACKNOWLEDGED -> OBSERVED On restart: PLANNED can be

I develop several tightly related repositories at the same time. Some are reusable SDKs for declarative schemas, infrastructure, stateful workflows, and other domain abstractions. Others are applications that consume several of those SDKs together. The development loop constantly crosses package boundaries. SDK A ──────┐ │ SDK B ──────┼──▶ application │ │ SDK C ──────┘ │ ▲ │ └──── feedback ───┘ I've already written about why I don't think this requires a monorepo, and why I prefer the repository itself to carry the current source of truth: AI Agents Don't Need a Monorepo. They Need a Readable Codebase The Repo Is the Context: Why Agents Don't Need History I won't repeat those arguments here. This post starts one layer later. As these SDKs became more agent-aware, each package started needing to tell coding agents how it should be used. I was already using project-local surfaces such as .claude/ , .codex/ , AGENTS.md , and package-specific skills. They are useful. Explicit project-local context works. The maintenance was the awkward part. When an SDK changed, I would tell the agent to update the corresponding instructions, rules, or skills in the consuming repository. That worked too. But after doing it repeatedly across several packages and repositories, I noticed something: My repeated update instructions had quietly become an undocumented protocol. Which files should change? Which source is canonical? What should be copied? What should only be referenced? What belongs to the package, and what belongs to the consuming repository? How should different coding-agent harnesses receive the same package knowledge without creating independent copies? I initially thought I needed a better synchronizer. I now think the problem is one layer higher. Packages already have an interface for programs. They increasingly need an interface for coding agents. I've been calling the protocol I'm using for that interface AIC — Agent Index Convention . It is still a draft from my own dev

In the last couple of posts I let agents debug over DAP — breakpoints, step over, continue. That's real debugging. But it's only half of it. When I debug something for real, I also click : I press the button and watch what happens, read the dialog, notice the toggle is greyed out. No backtrace ever tells you the Save button never enabled. So — can the agent do that half too? The web is the easy case Browsers are automatable by design. Most agent tools ship their own browser or drive an external one; point Playwright at a page and every element has a stable, queryable handle. The DOM is an accessibility tree wearing a different hat — roles, labels, structure, all there for the reading. For the web, this half of debugging is close to solved. Native apps are another game There's no DOM. When the agent has nothing to go on, it falls back to the eyeball approach: take a screenshot, let the model look, maybe run OCR or a pre-analysis pass to label what's on screen. It works — and sometimes it's the only option — but it's brittle (a few pixels off and the click misses) and it burns tokens describing pictures. I ran into this by accident. I once wrote a tiny skill whose only job was to screenshot a running 4D form and stitch an animated GIF for a README — 4d-capture-gif . Then I noticed Claude Code reaching for it to debug : the skill also reports a bit of the form's structure — where the buttons are — so the agent knows where to click. For simple cases it genuinely works. But screenshots-plus-coordinates is not the thing I want to build on. The cleaner path: read the tree, don't look at pixels Instead of staring at the screen, read the UI tree directly. On macOS you can script the Accessibility API from Python (pyobjc), and there are automation libraries to help. Now you're clicking element #37, the "Save" button instead of coordinate (412, 260), and hope . A couple of open-source tools are pushing exactly here: agent-desktop — a native CLI that exposes any app's accessibi

I spend a lot of time in the AI space -- reading papers, building things, talking to engineers who are actually shipping. And there is a gap between what the demos show and what production systems actually look like that nobody is being fully honest about. So here is my honest take on where things actually are. The Problem With How We Talk About AI Agents Everyone is calling everything an "agent" right now. A function that calls a tool? Agent. A chatbot with memory? Agent. A script with a loop? Agent. This dilution is not just semantic. It is causing real engineering mistakes. When you do not have a precise definition for what you are building, you end up over-engineering simple pipelines and under-engineering genuinely complex ones. I have seen teams spend weeks adding "agentic" orchestration to workflows that would have been fine as a single well-structured prompt. Here is the definition I keep coming back to: an agent is a system that has an objective, not just an instruction. It decides what to do next. It handles failure. It knows when it is done. Everything else is just a fancy function call. 🟢 If your system needs a human to tell it each step, it is not an agent. It is a chat interface. 🔵 If your system can recover from a failed tool call and try a different approach, you are getting somewhere. ✅ If your system can decompose a goal into subtasks and delegate them, that is the real thing. What Is Actually Happening in Production Right Now The honest picture from teams I follow and talk to: Most real agent deployments are narrow. They do one thing well. Customer support triage. Document extraction. Code review on a specific codebase. They are not general-purpose reasoning engines. They are purpose-built pipelines with some intelligence in the decision layer. The teams getting good results are not chasing the latest model release. They are obsessing over: ☑️ Tool design -- what can the agent actually call, and how clean is the interface ☑️ Failure handling -- wh

AI agents don't usually become unreliable all at once. They degrade quietly. One session the agent...

Everyone demos a WhatsApp chatbot. Few run one in production with real customers sending real messages 24/7. After 18 months of running SARA — an open-source WhatsApp AI agent serving businesses across 20 industries — here's what we learned about architecture that survives contact with reality. Why WhatsApp? The numbers are simple: 2B+ monthly active users 60% of SMB customers prefer messaging over calling 98% open rate (vs 20% for email) But WhatsApp is NOT just another chat channel. It has unique constraints that break naive implementations. Architecture Overview WhatsApp (WAHA) → Bridge (:3008) → SARA API (:3006) → AI Provider Chain → Tool Dispatcher ↓ Groq → Cerebras → SambaNova → Mistral The Provider Fallback Chain Single-provider AI is a production risk. We use a 4-provider chain: Primary: Groq (fastest, free tier) ↓ fail Fallback 1: Cerebras ↓ fail Fallback 2: SambaNova ↓ fail Fallback 3: Mistral (paid, always works) Each provider gets 2 retries with exponential backoff before failover. Result: 99.7% uptime over 6 months with $0 inference cost (free tiers). Tool Calling: Not Just Chat SARA doesn't just answer questions. She executes actions: create_reservation — books a table with date normalization ("domani alle 8" → 2026-08-10T20:00) check_inventory — queries stock levels generate_invoice — creates a PDF from database records schedule_appointment — manages calendar slots The dispatcher maps 30+ tools to handlers with an autonomy gate: User message → Intent classification → Risk assessment → Tool execution ↓ Low risk: execute immediately Medium: execute + notify owner High: ask for confirmation first You do NOT want your AI agent booking a catering order for 500 people without human approval. PII Handling Messages contain names, phone numbers, addresses. Our pipeline: Anonymize before sending to LLM (replace "Mario Rossi" → "[PERSON_1]") Process with anonymized data De-anonymize tool calls only (the reservation needs the real name) Never log PII in plain tex

AWS introduced something this week that is close enough to the problem I have been working on that I do not think it should be casually labeled complementary. Amazon Bedrock AgentCore added temporal policies , along with an open-source policy language called Dogwood . Instead of asking only whether an individual tool invocation is allowed, the gateway can evaluate the sequence of actions that led to it. Consider a purchasing agent with this rule: purchases under $10,000 do not require escalation The agent makes six purchases of $9,000. Every individual action satisfies the rule. The sequence may violate the organization's intended limit. The same problem appears with approvals. An API call may be permitted only if a human approval occurred earlier in the workflow. Looking only at the final call cannot establish that condition. Something needs to remember the relevant execution history and evaluate policy against it. That is the class of problem temporal policy addresses. The interesting architectural choice is that this logic lives outside the agent. The model does not need to faithfully remember the constraint from its prompt. The runtime owns the control. More agent behavior is becoming explicit This is not the only sign that agent instructions are moving out of conversations and into inspectable artifacts. A recent ESEM 2026 study of Agent Plans screened 36,710 engineered GitHub repositories and found 85 Markdown plan files across 10 repositories. That is a very small population, so I would not interpret the result as evidence of broad adoption. But the content is interesting. Those plans commonly described implementation steps, specific files or locations, and testing or validation instructions. The agent's execution intent was being preserved as part of the repository. There is a similar pattern in distribution. Tenable's CyberAgents Exchange treats agents, skills, MCP servers, and multi-agent playbooks as separate reusable components. The ecosystem is graduall

Our AI Agent Failed 5 Times in One Day. Here is Why It Never Happened Again. LAO Runtime Protection in action — real failures, self-repaired, permanently prevented, zero repeats. August 9, 2026 · by the ZWISERFIT engineering team AI agents fail silently. LAO makes failures visible and fixable. On August 8, 2026, our agent orchestration system — LAO — ran a full 24-hour cycle under autonomous governance. The result: 5 distinct failures detected, repaired, anchored, and permanently prevented across 3 agents (Shuyu, Luna, Hermes) in 5 different failure modes. Not one error repeated. Not once did a founder intervene in the repair loop. That is the claim. Here is the evidence. The Philosophy: Errors Dont Reduce Trust — Hiding Them Does 错误不会降低信任，隐藏错误才降低信任。 Errors dont reduce trust. Hidden errors do. This isnt motivational rhetoric. Its an engineering constraint. Every event in our trust ledger follows the same chain: failure → detection → repair → prevention → anchor An anchor is the key word. Not a bug report that gets archived. A persistent, versioned rule that makes the same class of error structurally impossible going forward. Anchors are the immune memory of the system. All metrics below are verified from ledger data. Error 1: Feishu Hallucination + Skill Amnesia An agent pushed a platform integration the founder never asked for, then forgot the corrected instruction entirely. Correcting an agent without persisting the correction fixes nothing. Repair: Three immutable anchors locked output standards. Intent Validation Gate v2 now blocks any non-requested platform integration before it is attempted. Error 2: Port Confusion — Knowing ≠ Executing An agent understood the right pattern but executed the wrong port — twice. Knowing and doing diverged. Repair: Structural prevention, not a better prompt. Error 3-5: URL mishaps, gate collisions, and silent failures The same class of mistake hit multiple agents independently. One gate stopped all of them. The Numbers Metric Val

TypeScript teams rarely standardize on one AI framework forever. One service may use Vercel AI SDK, another LangChain.js, another OpenAI Agents SDK, and a mature system may call provider clients directly. Those implementations expose different callback, telemetry, and streaming surfaces. Observability becomes expensive when every dashboard, test rule, and CI report understands each framework independently. An adapter layer isolates that variation. Framework-specific code captures source events; the adapter translates them into one versioned trace model; the rest of the system operates on normalized events. framework callbacks or wrappers | v framework adapter | v versioned trace events | | | v v v local UI CI gates telemetry export The goal is not to pretend every framework is identical. The goal is to preserve a common set of observable facts without leaking framework details into every consumer. Put the Boundary in the Right Place A tempting interface is runWithTrace(input) -> { result, events } . It works in a demo but creates several problems: Streaming runs may not have one finite completion point. Buffering every event in memory does not scale. Callback-driven frameworks already own the run lifecycle. A framework may emit activity after the initial method returns. Returning events couples capture, storage, and application results. A stronger boundary translates source events as they arrive and sends normalized events to the tracing core. Define the Normalized Model First Keep the shared model small, explicit, and versioned. type SpanKind = ' run ' | ' model ' | ' tool ' | ' retrieval ' | ' decision ' ; type TraceEvent = | { schemaVersion : 1 ; event : ' span_started ' ; traceId : string ; spanId : string ; parentSpanId : string | null ; name : string ; kind : SpanKind ; timestamp : string ; attributes : Record < string , string | number | boolean > ; } | { schemaVersion : 1 ; event : ' span_ended ' ; traceId : string ; spanId : string ; timestamp : string ; st

Large language models have enormous context windows now. That does not mean they use all of that context reliably. As prompts grow, models can miss details, lose track of relationships, or produce plausible summaries instead of doing the exhaustive work a question requires. The Recursive Language Models (RLM) paper proposes a different interface: keep the large context outside the model, expose it as a variable in a persistent programming environment, and let the model inspect, partition, and recursively query smaller pieces. We rebuilt that method with an unusual constraint: no OPENAI_API_KEY ; Codex CLI as the model backend; gpt-5.4-mini for both the RLM root and every subcall; a direct frontier model only as a separate baseline. The result was encouraging, expensive, and more nuanced than “cheap model equals frontier model.” What an RLM changes A normal model call looks roughly like this: large prompt -> model -> answer An RLM instead gives the root model metadata about the input and a Python REPL containing the real context: question | root model | persistent REPL holding the context |-- inspect and search with code |-- split context into useful chunks |-- call smaller LMs over those chunks |-- validate and aggregate results `-- return the final answer The important detail is that the root model does not need to carry every document, record, tool result, and partial answer in its own context window. Large intermediate values can remain in REPL variables. Subcalls receive focused, locally understandable tasks. That makes RLM less like a bigger prompt and more like an out-of-core data-processing system whose semantic operator happens to be a language model. What we actually tested We used an OOLONG trec_coarse validation example from the protocol described in the RLM work. The input was a 308,367-character context containing 3,182 general-knowledge questions. Each question implicitly belonged to one of six answer types: numeric value entity human being location ab

You're Not Comparing Models. You're Comparing Contracts. Two teams publish scores on the same agent benchmark. One lands in the low sixties. The other clears seventy. A procurement team reads the spread and makes a call. What they do not see: both teams may be running the same model. They did not need to change the weights for the gap to appear. The spread can come from scaffold alone. One team wrapped the model in a harness with better retries. Different tool defaults. A planner step the other team had skipped. None of that appears on the leaderboard. The comparison that drove the decision was not between two agents. It was between two contracts. There Is No Benchmark The mistake hiding behind this story is a category error. People talk about agent benchmarks as if they measure a thing called “the model.” They do not. They measure a coupled system. The model is one component. The rest is a stack of protocol decisions that are almost never disclosed and almost always matter. The score is the output of that stack. Change any layer and you change what the number means. Recent research on agent evaluation has named those layers explicitly. There are at least seven. Deployment regime. Observation channel. Harness and scaffold. Metric and action. Configured evaluator. Grader protocol. Audit bundle. Each is a contract. Each is negotiable. And each can silently change the verdict while the headline looks the same. That is what a benchmark actually is. Not a measurement of a model. A measurement of an entire testing contract, of which the model is one slot. There is structural reason the seven layers are the seven layers. They cluster into three corners that show up in almost every published agent-evaluation failure. What the model is rewarded for. How that reward is optimised. And how the test contract differs from production. Once you hold those three corners in view, the seven-layer stack stops feeling like a checklist and starts behaving like the actual shape of what is