A Framework-Agnostic Testing Methodology for AI Agents (61 sources, 58 test blocks, OWASP Agentic Top 10)

How do you actually test an AI agent? Not "does it respond," but: does it route to the right tool, chain calls correctly, recover from failure, resist prompt injection, and stay within cost/latency budget? I spent weeks working through this on a running agent, and open-sourced the entire methodology — framework-agnostic , so it applies regardless of your language, runtime, or toolset. What's inside •⁠ ⁠ 61-source benchmark map — BFCL, GAIA, τ-bench, SWE-bench, WebArena, AgentDojo, LongMemEval and more, categorized by what they actually measure •⁠ ⁠ 58 universal test blocks across 7 tiers (L1–L4, Error Recovery, Multi-Turn, Security). Each block = a tool-agnostic capability definition + a concrete reference implementation •⁠ ⁠ Full OWASP Top 10 for Agentic Applications 2026 (ASI01–ASI10) mapped to 6 universal security test blocks •⁠ ⁠ Evaluation methodology — LLM-as-Judge biases, pass@k vs pass^k, trajectory vs end-state, observability (OpenTelemetry GenAI), automated red-teaming (garak, PyRIT, DeepTeam) •⁠ ⁠ Regulatory alignment — NIST AI RMF, MITRE ATLAS, EU AI Act, ISO/IEC 42001 How to use it Take Part II, replace the reference-implementation fields with your own agent's tool names and expected outputs. The universal capability definitions need no changes. Blank templates are included. PheronAgent (a macOS agent with 50+ native/MCP tools) is included as a real reference case study — but the methodology is the product, not the agent. No marketing narrative: ⁠ STORY.md ⁠ documents the real bugs, real test runs, and real corrections that shaped each version. Docs are CC BY 4.0, templates are MIT. Issues and PRs welcome. 👉 https://github.com/trgysvc/AgentTestMethodology