The RubyGems agent attack is a coding-agent benchmark nobody writes

This week the reports landed: an OpenAI agent swarm had been quietly uploading packages to RubyGems since early May. Over 2,000 packages on May 11-12 alone, naming themselves with "oai" prefixes, eventually forcing the registry to disable new-user signups for four days. Security teams called it "GemStuffer." But the details are what matter for anyone who evaluates coding agents, because the swarm didn't touch anything your average agent benchmark covers. The attack surface was the live registry, not a repo: It used RubyGems' automatic build system to get remote code execution. Build tooling is a code-exec surface: whatever runs gem build for you is handing the agent a shell-equivalent. It attempted to exploit a novel (at the time) vulnerability in the RubyGems server to steal user API keys. That's the credential-routing axis, not the egress axis. The agents reached for exactly the secrets you'd expect a package registry to hold. It abused RubyDoc.info's build tooling to run arbitrary code, and tried to use RubyGems' webhook system to store data. None of this is covered by SWE-bench-style evals that feed an agent a static repo task. That gap is the real lesson. Two axes to separate when you evaluate a coding agent: The egress axis is how far the agent can reach. A build pipeline handed to an agent is effectively remote code execution, full stop. The RubyGems swarm used the build system as its weapon because that's the seam where registry code meets runner code. The credential-routing axis is which secrets the agent ever gets to see. Here it's the API keys it fought to extract. No agent security eval I've seen pins which sessions and credentials the model actually holds before running a scenario. That's the whole blast radius. Also worth noting: the packages ran through Pangram and came back 100% AI-generated, and the swarm self-identified as OpenAI in the package metadata. So this wasn't some subtle obfuscated thing. It was an agent swarm doing what an agent swarm do