HarnessDev: Enabling LLMs to Build and Iterate Agent Harness Systems

Abstract Agent harness serves as the core runtime control layer for large‑model‑driven agents. It defines execution loops, context management, state persistence, lifecycle handling and result verification logic, directly determining whether an agent can complete complex real‑world tasks reliably. Traditional agent development relies heavily on manual coding and human tuning of harness components, which brings heavy engineering overhead. HarnessDev, a joint research project by ByteDance Seed team together with multiple universities, explores a new research question: can large language models construct complete agent harness implementations and continuously revise these harnesses based on runtime feedback from downstream tasks. HarnessDev splits the full workflow into two major phases: Creation and Evolution. In the Creation phase, LLMs build runnable harness artifacts starting from a minimal weak seed harness. In the Evolution phase, the already‑generated harness receives runtime feedback, conducts iterative modification, and gets evaluated on unseen tasks. Researchers tested six different creator LLMs, covering four task domains, five benchmark suites and a total of 2027 downstream task instances. The experimental results reveal that modern LLMs are capable of generating functional harness code. However, many logical modules written by LLMs remain inactive in real execution. Portability across different executor models and runtime token overhead also become critical constraints for practical deployment. When integrating multi‑model workloads, developers may leverage an API gateway such as 4sapi to standardize model invocation traffic. 1. Background of Agent Harness Research Most existing agent benchmarks focus on evaluating task‑solving capabilities of agents. SWE‑Bench, Terminal‑Bench and other mainstream test suites usually adopt fixed pre‑written harness code. The harness handles environment interaction, tool invocation and output parsing, while the LLM acts pure