AutoAgent: Autonomous Harness Engineering
Kevin Gu · Founder, Thirdlayer
Today we are releasing AutoAgent, an open-source system for autonomously optimizing agent harnesses.
Given a task distribution and evals, AutoAgent runs a meta-agent that modifies the task agent's prompts, tools, verification, context management, and orchestration. It evaluates each change, analyzes the resulting trajectories, keeps improvements, and continues iterating.
After more than 24 hours of optimization, AutoAgent reached 96.5% on SpreadsheetBench and 55.1% on TerminalBench with GPT-5, ranking first on both leaderboards.
Motivation
Model capability is only one component of agent performance. The harness determines how the model interacts with its environment: which tools it can use, how context is presented, how work is verified, and how execution is orchestrated.
These choices are usually tuned manually and often differ substantially across domains. A strong spreadsheet agent requires a different harness from a strong terminal agent, even when both use the same underlying model.
AutoAgent treats harness design as an optimization problem.
The AutoAgent setup
AutoAgent separates the system into two agents with different objectives.
The task agent solves benchmark tasks. Its harness lives in agent.py, which contains the system prompt, tools, agent configuration, and orchestration.
The meta-agent improves this harness. Its research objective and constraints are specified in program.md.
A new run starts from a deliberately minimal task agent: GPT-5, a short system prompt, a 30-turn budget, and a shell tool. The meta-agent establishes a baseline and then repeatedly:
- runs the task agent on evals,
- reads trajectories and verifier outputs,
- identifies recurring failure modes,
- modifies the harness,
- evaluates the new configuration,
- keeps improvements and reverts regressions.
The primary objective is the number of passing tasks. At equal performance, AutoAgent prefers the simpler harness.
We also explicitly constrain the optimizer against task-specific solutions. A harness change should remain useful if the particular benchmark task that motivated it were removed.
Optimizing the whole harness
The search space is intentionally broader than prompt optimization.
AutoAgent can change tool definitions, prompts, turn allocation, execution flow, verification, subagents, and handoffs. This allows the optimizer to change the structure of the agent rather than search over a fixed set of parameters.
Across our runs, AutoAgent independently discovered several patterns that are common in high-performing manually engineered agents:
Specialized tools. It introduced structured tools for recurring operations where a generic shell interface created unnecessary failure modes.
Verification loops. It added explicit checking and correction before submission, including additional turns reserved for verification.
Self-generated tests. It instructed the task agent to construct tests and intermediate checks before completing tasks.
Context management. It moved large intermediate outputs into files and retrieved them selectively when needed.
Orchestration. It introduced specialized subagents and handoffs on task distributions where decomposition improved performance.
None of these behaviors were required components of the final harness. They were proposed by the meta-agent and retained through evaluation.
Trajectories are critical for optimization
We found that optimization was substantially weaker when the meta-agent received scores without task trajectories.
Trajectories expose the information required for targeted harness changes: which tools were used, what information was gathered, where execution diverged, whether outputs were verified, and when the agent stopped.
This lets the meta-agent optimize from behavioral evidence rather than only the final reward. In practice, many successful harness changes originated from recurring patterns across failed trajectories.
Results
We ran AutoAgent for more than 24 hours on two different task distributions.
Both runs reached the highest score on their respective leaderboards.
The notable result is that the final harnesses were not manually designed. AutoAgent started from a simple agent and accumulated improvements through autonomous experimentation.
What we learned
Several findings were consistent across our experiments.
Separating optimization from execution helps. We tested having one agent both solve tasks and improve itself. The meta-agent and task-agent split performed better. The two roles operate over different contexts and objectives: one optimizes an individual trajectory, while the other optimizes performance across a distribution of trajectories.
The optimizer will overfit if allowed to. Meta-agents can discover benchmark-specific shortcuts just as conventional optimization procedures can. We found it useful to explicitly require every change to generalize beyond the task that motivated it.
Meta-agent quality matters. Many harness improvements are derived from the meta-agent's own understanding of tools and agent behavior. Better meta-agents produced better task-agent harnesses.
We also observed that same-model meta-agent and task-agent pairings outperformed some cross-model pairings. One hypothesis is that a model has a particularly useful prior over the behavior of another instance of itself, including how it interprets tools and instructions. We refer to this as model empathy, and plan to study the effect more systematically.
What comes next
The number of useful agent configurations grows quickly across models, tools, and task domains. We do not expect the best harnesses to be designed once and remain fixed. As models improve and workloads change, the harness should be able to adapt with them.
AutoAgent is an early step in that direction. Today, it optimizes a harness against a fixed task distribution. We are interested in systems that can continuously improve deployed agents, assemble the right tools and context for new tasks, and maintain large fleets of specialized agents with substantially less manual tuning.
We have open-sourced AutoAgent so others can run the same optimization loop on their own evals and task domains.
If this type of work interests you, please reach out here.