标签：#Agents

Harness looks cool, yeah! I know its origin, its role in Testing, and why. But that's exactly what throws you off, the story you're expected to defend. There's something deeper. I opened my session with "hi", expecting my forced load via CLAUDE.md and my contract as always, and today, out of nowhere, the model suggested two services that needed my authorisation. Microsoft 365 and Zapier. I don't have, and never wanted, them authorised. I never asked for them. And here's the part that pisses me off: I went to check. And... I look on my machine and find nothing. No config, no credential, no trace. I look in the online settings and see them listed as suggestions, like the trending product (connector) of the moment sitting in the prime spot on a supermarket shelf, with a button that says Connect. There was no button to remove. There was nothing to remove. They had never been connected to anything. It was a storefront. And on top of that, the model was biased by injected instructions, in this case system-reminders steering behavior. The fucking little word The software that sits between you and the model, they call it harness. Sounds like something subtle, that helps... that improves things, that doesn't think. The word is partly right, it does extend what's called "inference" and it inserts itself right in the middle, opaquely, in the back-and-forth between APIs, MCPs, and the vendor's logic. No tech jargon You write a letter, put it in the envelope, drop it in the mailbox. On the way, someone opens it and slips in three more pages. Same handwriting. Same paper. Unsigned. Whoever receives it swallows it whole as if it were your original letter. That's exactly this. Your instructions and the vendor's arrive at the model through the same channel, mixed together, unsigned and unsealed. Nothing says who wrote what. That's "hardness", nothing more, nothing less... Sounds so modern in meetings. Like you know what you're talking about... It's a multi-factor fight I have instru

Previously: 9 Bugs That All Looked Like a Working System · I Built an AI That Rewrites Its Own...

Your coding agent is good at reading code. Point Claude Code or Cursor at a repo and it will figure out the language, the framework, the build command — it just costs you tokens and a few tool calls every session to re-derive what it forgot. What it can't read is the part that isn't in the code: who the project is for,and why it exists. So it guesses. Confidently, in the same tone it uses for the facts it actually verified. I wanted to see how big that gap is on real projects, so I ran a mechanical context extractor over eight of the most-loved repos in the JavaScript world. The method faf git <url> clones a repo and fills in a small typed context file ( project.faf ) from what it can find — README, package.json , project structure, config. No hand-authoring, no LLM writing prose. It fills what's there and leaves the rest blank. Nine-ish slots: the identity (name, goal, language) and the six W's — who, what, why, where, when, how. Run it yourself: npx faf-cli git https://github.com/facebook/react The result repo extracted who why facebook/react 56% — blank — — blank — vercel/next.js 44% — blank — — blank — expressjs/express 50% — blank — — blank — colinhacks/zod 67% — blank — — blank — sveltejs/svelte 88% — blank — — blank — prettier/prettier 75% — blank — — blank — Eight repos in total (React, Next.js, Express, Zod, Hono, Svelte, Vue, Prettier). Every one of them: who is this for and why does this exist came back empty. Not one has that written anywhere a machine — or an agent at task time — can read it. Svelte scored 88%. The who and why are still blank. This isn't a documentation-quality problem, and it's not a knock on any of these projects. The stack lives in the files. The intent lives in maintainers' heads, design docs, old RFC threads, and Discord history — none of which your agent has open when it's editing a file. Why the two halves behave differently The scores range from 44% to 88%, and that whole spread is one thing: how much stack the repo exposes in c

How do you ship a software product with 0 lines of manually-written code? A friend asked me this today, and I realized I didn't have a simple answer. So I dug deeper. It turns out the answer is in how you engineer your harness. Wait now, what? What is harness engineering? There is a reason this is the most important trend right now around coding agents. The biggest question these days is how to validate AI-generated code without reading every single line. How do you make sure an agent doesn't break production or delete your data? A blog by OpenAI shared an interesting experiment where a team of 3 engineers have built and shipped an internal beta of a software product with 0 lines of manually-written code. Every line of code: application logic, tests, CI configuration, documentation, observability, and internal tooling, has been written by Codex. How did they do it? They didn't write the app. They designed the harness. What exactly is a harness? Think of an AI agent like a powerful racehorse. The harness is the track, the blinders, and the jockey's reins that keep it running in the right direction instead of jumping into the stands. As my colleague Arthur Thompson explained today: for agents — the harness is composed of all the deterministic components that wrap the LLM. Balaji Subramaniam details those deterministic components in his blog — the orchestration layer, execution sandboxing, state persistence, and verification tools. If you want to build reliable agentic systems, your job shifts from writing the logic to designing the environment. Here is what you need to focus on: Set strict boundaries: Don't let the agent guess what it can touch. Enforce strict access rules (like confining it to a specific sandbox) so it can't accidentally wipe out production data. Build "Repair Loops": Agents will inevitably make mistakes. A great harness automatically traps errors, like a failed build or a test failure, and feeds those clean logs right back to the agent so it can fix

Previously: 9 Bugs That All Looked Like a Working System · I Built an AI That Rewrites Its Own Prompts · The Edit That Fixed 4 Tasks and Broke 1 · The Gate Is the Product · The Doctor Who Diagnosed Every Patient · 4 Models, 0 Promotable Edits In v0.1.0, an edit fixed 4 tasks and broke 1. Net +3 on 26 tasks. p=0.23. Gate rejected. The ceiling was clear: if you do not move enough tasks, the gate should say no. In v0.2.0, we expanded the A/B corpus to 40 tasks. We fixed the pipeline bugs. We added rejection context. We tested stronger models. The math got cleaner, not kinder. The ceiling shifted. It did not disappear. The v0.1.0 Result: 26 Tasks, 5 Movable, p=0.23 The edit was real. It fixed 4 tasks and broke 1: Task Prompt A Prompt B Expected Change classify-015 technical urgent urgent FIXED classify-023 security urgent, security urgent, security FIXED classify-024 feature feature, billing feature, billing FIXED classify-029 feature other other FIXED classify-014 technical feature technical BROKEN Net: +3. Mean delta = 0.115. p=0.23. The permutation test computes this by shuffling task labels 1,000 times and counting how often random chance produces a delta ≥ 0.115. 23% of the time — above the 5% threshold. The sign-test floor with 5 discordant pairs out of 26 is ~0.031 one-sided. Even a flawless edit that fixed all 5 would barely clear p<0.05 two-sided. The v0.2.0 Result: 40 Tasks, Still Nothing We expanded to 40 tasks. We should have more power. Here's what Mistral 24B, our strongest analyzer, produced: Iter p-value Mean delta Accuracy 1 0.55 +0.025 64% 2 0.52 +0.025 64% 3 1.0 0.0 64% 4 0.52 +0.025 64% 5 0.77 -0.025 64% Mistral produced positive deltas in 3 of 5 iterations. That is real signal. But the delta is +0.025 — 2.5% improvement on 40 tasks. At p~0.5, there is roughly a coin-flip chance this is noise. The ceiling did not disappear. It moved: with 40 tasks, the sign-test floor for a flawless edit that moves 5 tasks is ~0.016 one-sided — clearable. But Mistral

A coding agent can write syntactically plausible code and still leave a broken project behind. The obvious answer is to give the agent an lsp tool and let the model ask for diagnostics whenever it wants. SolonCode tried that shape first. The implementation put navigation and diagnostics in one tool, but diagnostics were effectively never requested. That result is not surprising: after a write, “check whether this introduced errors” is not an optional curiosity. It is part of the write operation’s feedback loop. SolonCode’s current design makes that distinction explicit: write and edit trigger diagnostics automatically after a successful change. read warms the language server asynchronously without delaying the read. The lsp tool is reserved for optional navigation such as definition, references, hover, symbols, and call hierarchy. The interesting engineering is not starting a language server. It is keeping the file, the language server, the model, and the Web UI consistent while all four observe different representations of the same change. Diagnostics should follow a write, not a model decision The implementation note in the repository describes the original failure plainly: ten capabilities—nine navigation operations plus diagnostics—were exposed through one tool, and diagnostics were “never called” in practice. That led to a three-layer design: write / edit / apply_patch -> sync the file -> wait for diagnostics -> append diagnostics to the tool output read -> warm up the language server asynchronously -> do not wait and do not change the read result lsp -> definition / references / hover / symbols / call hierarchy ... This is a useful rule for agent design: feedback that is necessary to evaluate a mutation belongs on the mutation path. Exploratory information can remain an explicit tool. The separation also keeps the tool schema smaller and the model’s decision burden clearer. The model does not need to remember a second call after every edit just to discover whe

I spend a lot of time in the AI space -- reading papers, building things, talking to engineers who are actually shipping. And there is a gap between what the demos show and what production systems actually look like that nobody is being fully honest about. So here is my honest take on where things actually are. The Problem With How We Talk About AI Agents Everyone is calling everything an "agent" right now. A function that calls a tool? Agent. A chatbot with memory? Agent. A script with a loop? Agent. This dilution is not just semantic. It is causing real engineering mistakes. When you do not have a precise definition for what you are building, you end up over-engineering simple pipelines and under-engineering genuinely complex ones. I have seen teams spend weeks adding "agentic" orchestration to workflows that would have been fine as a single well-structured prompt. Here is the definition I keep coming back to: an agent is a system that has an objective, not just an instruction. It decides what to do next. It handles failure. It knows when it is done. Everything else is just a fancy function call. 🟢 If your system needs a human to tell it each step, it is not an agent. It is a chat interface. 🔵 If your system can recover from a failed tool call and try a different approach, you are getting somewhere. ✅ If your system can decompose a goal into subtasks and delegate them, that is the real thing. What Is Actually Happening in Production Right Now The honest picture from teams I follow and talk to: Most real agent deployments are narrow. They do one thing well. Customer support triage. Document extraction. Code review on a specific codebase. They are not general-purpose reasoning engines. They are purpose-built pipelines with some intelligence in the decision layer. The teams getting good results are not chasing the latest model release. They are obsessing over: ☑️ Tool design -- what can the agent actually call, and how clean is the interface ☑️ Failure handling -- wh

Originally published on tamiz.pro . We have collectively lost our minds over benchmarks. AgenticBench scores 90%? Great. Multi-Agent Hallucination Leaderboard rank #1? Impressive. Yet the moment you ship that same agent to a chaotic production environment with 14,000 SQL dialects, flaky APIs, and users who refuse to follow instructions, it collapses within hours. This is not a bug. It is a feature of how we evaluate these systems. The gap between benchmark performance and production reliability is the single most dangerous illusion in current AI engineering. Benchmarks measure capability ; production measures consequence . If you are building AI agents today, you are likely flying blind. Here is why your evaluation strategy is lying to you, and what actually happens when agents hit the wire. The Snapshot Fallacy Benchmarks are snapshots. They are static, curated, and deterministic. An agent tasked with answering a question from Wikipedia is doing retrieval and generation. In production, that same agent might be triggering a refund API while concurrently writing to a database. The problem is that benchmarks rarely account for statefulness . A chatbot that generates a perfect summary is qualitatively different from an agent that executes a five-step workflow where Step 3 depends on the output of Step 1, which was corrupted by a non-deterministic tool response in Step 2. Benchmarks usually test the trajectory in isolation. They do not test the persistence of the state across 10,000 concurrent requests. When you move from benchmark to production, you introduce temporal decay . The model context window fills. Tool schemas drift because the upstream API changed yesterday. Database schemas evolve. The agent you tested in January is functionally a different entity in June, yet your evaluation suite remains frozen in time. The Tooling Cliff The most common failure mode in production agents is not hallucination—it is tool failure . In a benchmark, if you ask an LLM to get_wea

Here's the moment a lot of us had this year. You built an agent. It was genuinely impressive — it...

Stop drawing the graph: reactive agents over versioned artifacts Most agent frameworks make you draw the graph : connect nodes, wire memory, declare control flow. But a knowledge problem is not a workflow. Take a realistic question: "Why did infrastructure costs increase in Q2?" The answer may need Confluence docs, GitLab merge requests, CSV spend data, a calculation, source verification — and a clarifying question. The next question needs a different path. There is no universal graph here, and asking a developer to draw one for every possible question is asking them to predict the future. So we built an agent runtime where you don't describe execution at all . You describe what artifacts exist and what agents can do with them; the runtime derives what runs next from state changes. Agents react to events. There is no graph and no node pipeline. This is ctxloom — a reactive, artifact-driven agent runtime, now open source. What it looks like The whole loop is: create an artifact → agents react → one atomic patch → context advances . A knowledge question — say, "how much does GPU inference cost?" — becomes a chain of typed artifacts: UserQuery → TypedDoc → Evidence → Claim → Answer . Each is produced by an agent that reacts to the previous artifact. No graph describes this chain; it falls out of what each agent consumes and produces. ARTIFACT CREATED / UPDATED │ ▼ AGENTS REACT ──self.effects──► Effects ──compile──► Patch ▲ │ └──────────────────────────────────────────────────────┘ Context v+1 The event that wakes an agent is derived from that same change — the causal chain can never drift from the actual state. from pydantic import BaseModel from ctxloom import Budget , Consume , Context , Runtime , RuntimeResources , create_agent , produce , structured_llm class Question ( BaseModel ): text : str class FindingBody ( BaseModel ): text : str class Finding ( BaseModel ): text : str source : str class Conclusion ( BaseModel ): text : str @produce ( Finding ) async def ana

OpenClaw has released OpenClaw 2.0, a major update to the open-source personal AI agent that changes its installation process, browser interface, memory, skills, automations, plugins, security, and collaboration features. By Daniel Dominguez

LLM LLM is a model, which means an equation. Example: y = mx + c y = m1x^3 + m2x^2 + m3x + m4 A model is actually made up of weights . In any model, e.g., ChatGPT model or Gemini model, they would have used a large amount of input to train the model. Input means a large amount of text/image data that is available on the internet. The input would have been fed into the Transformer architecture to get the output, which is the model. Weights are floating-point numbers that represent the model's learned parameters. A 10B or 100B parameter model means how many parameters (weights) are present inside the model. We cannot store a large-parameter model on our computer due to inadequate storage and computational power. Storage and CPU/GPU power decide what size of model can be run on a computer. To run a model locally, we can use one of the following tools: Llama.cpp Ollama LM Studio Open Weight Model vs Open Source Model An open-weight model shares its model weights. So, we can run them, fine-tune them, and host them on a local system. Here, the training code, data, and full methodology are not shared. Whereas, in an open-source model , the weights, training code, data, and sometimes the dataset are shared. Why Do We Need to Use LLMs? LLM is a next-word predictor . Suppose we ask: "Hi, how..." The answer can be: How are you? How do you do? How is your life? etc. These are possibilities. Here, most of the time, the answer will be "How are you?" because if a word has more presence, it has a higher possibility of occurring. Each possibility will have a score between 0 and 1 . We have 3 controlling parameters to control the output generated by the LLM. 1. Temperature Usually set from 0–1 . It controls the randomness of the model. If the value is 0–0.3 , which is low, it means generating the most likely words, i.e., facts or commonly occurring words. If the value is high, the model will choose less likely words. We use this high value in storytelling and creative writing . 2. To

AIR's platform can discover agents running at a company, continuously vets any skills and add-ons they use, and blocks any unwanted behaviour.

The Brave Wanderer: I made Claude play a Pokémon it never read a guide for Full timeline video of this 2,000-turn run (game frames + a live cost counter on the left, the model's real-time thinking log on the right): https://youtu.be/ewyM7mzGzTM At the end of the first article in this series , I made a promise. Fable 5's fluency in FireRed owed half its credit to the walkthroughs it had memorized — it wrote down "Oak's Parcel," an item the game hadn't shown it yet, 141 turns early. So the only honest exam is a new exam paper: "Same harness, same model, a map it cannot recite — I'll post the numbers." This article is those numbers. The exam paper is Pokémon Team Rocket Edition — the Chinese fan translation of the Spanish community hack Pokémon Edición Team Rocket, released in January 2026. You play a Team Rocket recruit working your way up from the Five Island base. Five story rounds, four regions; the Kanto chapter alone is labeled 30-35 hours for a human player. And most importantly: this game is essentially absent from the model's training data . No guide to recite. Just the screen and itself. There's also a lovely narrative twist: the hack sets your home base inside the original FireRed's Five Island Rocket Warehouse — the enemy hideout you raid late-game as the hero in the official version. Same map, opposite allegiance. Rules unchanged: vision only, one screenshot plus its own notes per turn, one button-press tool, a 2,000-action cap. The result, up front 8 hours 43 minutes, 2,000 turns, $113.44. It reached the middle of the prologue's first mission — roughly 40-60 minutes of human play time. It taught itself plenty: menus, battles, catching, the save flow, all from scratch; after losing to a fellow recruit it wrote a revenge battle plan into its notes, ground levels, and actually won the rematch; it even induced map rules like "dark blue water can't be surfed, light blue can," and maintained a dead-ends list and an NPC-interview checklist in its notes. One deta

Fambot is building an AI “chief of staff” to help families manage the emails, calendars, school updates, sports schedules, and other logistics of raising kids.

A real model's write, escrowed before it landed What I can honestly claim here, and only this: I put an escrow membrane in front of a real OpenClaw gateway as a before_tool_call plugin, watched a real LLM's tool call go through it, and confirmed the whole loop end to end, escrow, admit, commit, undo, with a byte-for-byte restore. That's it. I've read 0.077 percent of the OpenClaw source (29 of 37,659 files, counted cumulatively across three separate rounds of this work), the Escalate branch has never fired in a real run, and I haven't found one confirmed example of a ClawHub-distributed plugin using this hook. None of that changes what happened on 2026-09-01. All of it belongs in the same paragraph as the claim, not three screens down where nobody reads it. The project behind this is gx (TraceFold, Apache-2.0), a layer that treats every effect an agent produces as something you escrow, gate, and can invert, rather than something you log after the fact and hope you can undo by hand. OpenClaw is steipete and vincentkoc's agent runtime, MIT-licensed, npm-distributed (204.8 MB unpacked at 2026.8.1), with a plugin hook called before_tool_call that fires before a tool's execute() runs and can block it outright. Four ways to fail before you fail correctly I want to write the failures first, because they're the part that actually shows how the system works. The id was a filename, not an identifier. gx writes receipts to disk with underscores in the filename ( gx1_smxcmcdm...json ), because colons aren't safe in filenames on every platform. The identifier gx undo actually wants uses colons ( gx1:smxcmcdm... ). I copied the filename straight into the undo command and got VALIDATION_ERROR: not a gx1: id . The right string was sitting in an index file two directories over. I hadn't checked. I trusted --offline to mean less than it means. I wanted a verification that touched nothing but the receipt itself, so I ran gx receipt verify --offline --project <bed> and got back valid:f

AgentSelfEdit is an open-source sidecar that rewrites its own system prompt from execution feedback....

Dear past Jenna, I know you're used to dealing with large, complex, legacy codebases riddled with...

AgentSelfEdit is an open-source sidecar that rewrites its own system prompt from execution feedback. It A/B tests edits and promotes only statistically-proven winners. Code: github.com/deghosal-2026/agent-self-edit I built an AI that rewrites its own prompts. It looked like it worked. It didn't. Over a single session, I found and fixed 31 issues. Nine of them were fundamental — each one made the system look like it was working when it wasn't. The most dangerous was this: the promotion gate was letting noise through as "improvement." It checked p < 0.95 instead of p < 0.05 . Almost everything passed. A "promotion" at p=0.1 had a 10% chance of being random noise. Two lines of code separated a system that learns from a system that drifts. But that wasn't the only one. The A/B test "passed" because it compared a prompt against itself. The scoring "passed" because it accepted any non-empty response. The Docker test "passed" because it skipped the hard parts. The failure traces were fabricated. The gate received the wrong prompt. The CLI talked to a mock instead of a real LLM. The config silently ignored the endpoint. And the field test runner was measuring the wrong thing entirely. The most dangerous bugs aren't the ones that crash. They're the ones that produce output that looks correct. The system always produces something — the question is whether that something is real. Here's every bug, how it hid, how I caught it, and what it taught me about building systems on top of LLMs. Bug 1: The Gate Was Letting Noise Through as "Improvement" This was the most insidious bug. The gate promoted an edit. Accuracy jumped from 20% to 40%. I celebrated. Then I looked at the code. # What was written: passed = p < confidence_level # p < 0.95 # What it should have been: alpha = 1 - confidence_level # 0.05 passed = p < alpha # p < 0.05 The gate was checking p < 0.95 instead of p < 0.05 . Almost everything passed. A p-value of 0.9 would pass. Even 0.5 would pass. The "promotion" at p=0.

Every agent framework README says "multi-agent". AutoGen, CrewAI, LangGraph, MetaGPT each have 20k+ GitHub stars, and the term "multi-agent" appears in thousands of repo descriptions. But what do projects that call themselves multi-agent actually implement? Until now, nobody measured the population — only how to build frameworks, or theory about whether multi-agent is "just prompting". A new census (86 strictly-filtered, self-described "multi-agent" repos with 1k+ stars, plus 18 seed frameworks, snapshot-pinned) annotates a three-axis taxonomy to full-population ground truth: (i) model-instance structure, (ii) topology, (iii) judge/critic presence. The headline: the label-reality gap 68.2% of self-described multi-agent repos (58/85, Wilson 95% CI [57.7%, 77.2%]) are single-model or non-agent systems. "Multi-agent" overclaims what is implemented. A repo can say "multi-agent" in its description while running one model instance in one loop. Among the 27 genuine multi-agent systems: Orchestrator-worker is the plurality topology — 48.1% (13/27, CI [30.7%, 66.0%]) — coordinator + workers, not peer teams. Judge/critic agents are rare — 1 of 30 annotated repos (3.3%). For all the talk about critic/reviewer agents, almost nobody implements them. There's also a reverse gap: monorepo-aware manifest extraction found 44 repos with framework dependencies — including repos that use multi-agent frameworks (langroid, lumibot, wigolo...) without claiming the label. The gap runs both directions. Why the census matters (and its honest limits) The classifier went through three documented generations: v1 degenerate, v2 framework-API (81.2% full-population), v3 README-role (100.0% in-sample, mechanistic rules, no repo-name hardcoding). Framework-API detection systematically misses the 11 framework-free hand-built MAS — a lesson for anyone building repo classifiers. The primary axis (86 repos) is full-population human-annotated with a 2-pass re-verification protocol. Honest disclosure in t