An Empirical Study of Harness Design for Coding Agents
Recorded: Sept. 18, 2026, 2 p.m.
| Original | Summarized |
[2609.20804] An Empirical Study of Harness Design for Coding Agents
Skip to main content Search Log in Search arXiv Press Enter to search · Advanced search Computer Science > Artificial Intelligence arXiv:2609.20804 (cs) [Submitted on 17 Sep 2026] Abstract:Coding harnesses shape how autonomous coding agents translate model capabilities into long-horizon software-engineering performance, yet existing work typically evaluates harnesses as monolithic systems, leaving the effectiveness of individual components unclear. To enable component-level comparisons, we study this question with a lightweight coding harness whose execution loop is fixed while three components are varied: planning, action space, and context management. Across four models evaluated on SWE-Bench Verified and Terminal-Bench 2.1, we evaluate 176 matched settings spanning five context-management strategies, four context-window budgets, and targeted ablations of planning and action space. We find that: (1) Context management becomes increasingly valuable as the context-window budget tightens, with most of its benefit coming from preventing context-overflow failures. (2) Staging rule-based elision before LLM-based summarization provides the strongest overall efficiency among the context-management strategies, whereas making elided content recoverable adds machinery that models rarely use and yields no accuracy gain. (3) Planning shifts from an accuracy scaffold for weaker models to a cost saver for stronger models, with little change in accuracy. (4) Predefined tools improve performance for models with weaker bash proficiency, whereas bash-capable models can operate effectively with a bash-only interface and achieve substantially lower cost, especially on command-line-centric tasks. Trajectory-level analysis explains these effects: context management extends execution trajectories without substantially altering agent behavior, planning changes where trajectories stop, and the action space changes the granularity at which code is written. These findings inform model- and budget-aware harness design and provide a modular framework for evaluating future harness components. Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG); Software Engineering (cs.SE) Cite as: Focus to learn more arXiv-issued DOI via DataCite (pending registration) Submission history From: Run-Ze Fan [view email] [v1]
Full-text links: View a PDF of the paper titled An Empirical Study of Harness Design for Coding Agents, by Run-Ze Fan and 8 other authorsView PDFHTML (experimental)TeX Source < prev | new Change to browse by: References & Citations NASA ADSGoogle Scholar export BibTeX citation BibTeX formatted citation loading... Data provided by: Bookmark
Bibliographic Tools Bibliographic and Citation Tools Bibliographic Explorer Toggle Bibliographic Explorer (What is the Explorer?) Connected Papers Toggle Connected Papers (What is Connected Papers?) Litmaps Toggle Litmaps (What is Litmaps?) scite.ai Toggle scite Smart Citations (What are Smart Citations?) Code, Data, Media Code, Data and Media Associated with this Article alphaXiv Toggle alphaXiv (What is alphaXiv?) Links to Code Toggle CatalyzeX Code Finder for Papers (What is CatalyzeX?) DagsHub Toggle DagsHub (What is DagsHub?) GotitPub Toggle Gotit.pub (What is GotitPub?) Huggingface Toggle Hugging Face (What is Huggingface?) ScienceCast Toggle ScienceCast (What is ScienceCast?) Demos Demos Replicate Toggle Replicate (What is Replicate?) Spaces Toggle Hugging Face Spaces (What is Spaces?) Spaces Toggle TXYZ.AI (What is TXYZ.AI?) Related Papers Recommenders and Search Tools Link to Influence Flower Influence Flower (What are Influence Flowers?) Core recommender toggle CORE Recommender (What is CORE?) Author About arXivLabs arXivLabs: experimental projects with community collaborators Which authors of this paper are endorsers? | We gratefully acknowledge support from About Major funding support from |
Coding harnesses are instrumental in determining how autonomous coding agents successfully translate the inherent capabilities of underlying models into effective long-horizon software-engineering performance. However, existing research often evaluates these harnesses as monolithic entities, which obscures the efficacy of individual components. To address this gap, the authors investigated the design of a lightweight coding harness by decoupling three key components—planning, action space, and context management—while fixing the execution loop. This empirical study was conducted across four distinct models evaluated on the SWE-Bench Verified and Terminal-Bench 2.1 benchmarks, analyzing 176 matched settings that explored different context-management strategies, context-window budgets, and targeted ablations of planning and action space. The analysis yielded several critical insights regarding component interaction and efficiency. Context management emerged as increasingly valuable as context-window budgets were constrained, primarily demonstrating its utility in preventing context-overflow failures. Furthermore, the study assessed various context-management strategies, revealing that staging rule-based elision prior to LLM-based summarization offered the highest overall efficiency among the tested methods. Conversely, incorporating additional machinery to make elided content recoverable was found to introduce overhead without providing any measurable gain in accuracy for the models. Regarding the planning component, it was observed that its role shifts depending on the model's proficiency: it functions as an accuracy scaffold for weaker models but transitions into a cost saver for stronger models, with minimal impact on the resulting accuracy. The study also investigated the action space configuration. Predefined tools improved performance specifically for models exhibiting weaker proficiency in bash scripting. In contrast, models already capable of bash operations demonstrated effective performance when utilizing a bash-only interface, achieving substantially lower computational costs, particularly for tasks centered around the command line. Trajectory-level analysis provided a deeper explanation for these observed effects. Context management was shown to successfully extend execution trajectories without altering the agent’s fundamental behavior. Planning modifications correlated precisely with where execution trajectories concluded, while changes to the action space affected the granularity with which the code was generated. These trajectory-level findings collectively inform a modular framework for harness evaluation and suggest principles guiding model- and budget-aware harness design. |