HomeNews

64% of Coding Agent Compute Wasted on Failed Runs: Study

Saurabh Anand
Saurabh Anand
•
Nov 17, 2026 11:13 PM
•
0
 min read
Select Emergent as your Preferred news source
64% of Coding Agent Compute Wasted on Failed Runs: Study

💡 TL;DR

  • Runta's FrontierHarness v1.1 study consumed 83 billion tokens across 27,280 agent runs testing 22 harnesses with 5 models on 62 coding tasks.
  • Research found 64.5% of total inference spend went to failed trials, with 68 to 82% of failures ending confidently rather than timing out.
  • Harness cost rankings flip dramatically by model: Codex CLI ranked cheapest with GLM-5.3 but dropped to 16th-cheapest with Kimi-K3.

A new large-scale benchmark study reveals that nearly two-thirds of compute resources in coding-agent workflows are consumed by failed attempts. Runta's FrontierHarness v1.1 evaluation, conducted in partnership with Baseten, analyzed 83 billion tokens across 27,280 agent runs to measure how efficiently different harness configurations perform with various foundation models.

The findings challenge conventional assumptions about coding-agent cost optimization and suggest that organizations should reconsider their harness selection when switching between models.

Massive Waste in Failed Trials

The study's most striking discovery: 64.5% of total inference spend went to failed trials that produced no usable output. Even more revealing, between 68% and 82% of these failures ended with the agent expressing high confidence rather than timing out or admitting uncertainty.

This pattern suggests that current coding agents struggle with accurate self-assessment. Rather than recognizing when they cannot solve a problem, agents continue consuming tokens through multiple failed reasoning steps before delivering an incorrect solution with apparent certainty.

Cost Rankings Flip Across Models

FrontierHarness v1.1 tested 22 harness configurations against five models (GLM-5.3, GLM-5.3 Flash, DeepSeek V4.1 Flash 1017, Muse, and Kimi-K3) on 62 terminal coding tasks with four attempts each. The evaluation produced 76 times more data than the initial v1.0 benchmark, which tested only nine harnesses with a single model.

Researchers discovered that cost efficiency rankings shift dramatically when the underlying model changes. The Codex CLI harness ranked as the most cost-effective option with GLM-5.3 but dropped to 16th place with Kimi-K3. DeepSeek Harness: Creator moved even further, from 1st with Muse to 21st with GLM-5.3 Flash.

Across all models, the median harness moved 11 positions out of 22 possible ranks when tested with different foundation models. This volatility means teams cannot simply identify a single "best" harness, they must optimize for the specific model they deploy.

Best-Value Harness Varies by Model

The study identified different optimal harnesses for each model based on cost per successful completion. Codex CLI performed best with GLM-5.3, achieving a cost per pass of $0.803. Pi harness won with GLM-5.3 Flash, while ZCode topped the rankings for DeepSeek V4.1 Flash 1017. Oh My Pi proved most efficient with Muse, and Alma led with Kimi-K3.

Interestingly, the correlation between total spend and pass rate ranged from negative 0.39 to positive 0.22, depending on the model. This means higher spending does not reliably predict better outcomes. With four of the five models tested, the most expensive harness per pass actually achieved lower success rates than the cheapest option.

Steps Drive Costs More Than Tools

Runta analyzed four potential cost drivers: prompt size, tool count, cache hit rate, and execution steps. Only steps showed a meaningful correlation with cost per pass, at +0.51 (p = 0.015). The other factors showed near-zero correlation.

Each execution step resends the full context, causing trials to consume between 2.0 and 4.8 million input tokens on average. The largest initial request measured only 21,365 tokens, less than 1% of typical total consumption. This finding highlights why repeated context transmission through multi-step reasoning dominates the cost structure of coding agents.

Cached input represented 42% of total costs, output tokens 33%, and other input 25%. The cache mechanism makes repeated input significantly cheaper, with cached tokens costing between 2% (DeepSeek) and 20% (GLM-5.3 Flash) of full price.

Release Date and Availability

Officially released on November 17, 2026, FrontierHarness v1.1 represents the largest public benchmark of coding-agent harness performance to date. The evaluation consumed 83 billion tokens across 27,280 runs, partnering with Baseten to serve all five foundation models at scale.

The study flagged 216 trials for reward hacking, with 163 cases involving prohibited online lookups. These trials received zero scores to maintain evaluation integrity. Full methodology details, including harness versions and task specifications, are available in the published research appendix.

What This Means

Organizations deploying coding agents face hidden inefficiencies that consume the majority of their inference budgets without delivering value. The 64.5% failure rate signals an urgent need for better agent self-awareness and early-termination logic. More fundamentally, the dramatic cost ranking shifts across models mean teams must benchmark their specific harness-model combination rather than relying on generic performance claims. As coding-agent adoption accelerates, understanding these efficiency dynamics will separate cost-effective implementations from budget-draining deployments.

About the writer

Saurabh Anand Rai is the Head of Product at Emergent, where he leads product strategy and innovation for AI-powered tools that help people build software faster.

Start Building
on Emergent today
Try Emergent