When teams compare AI agents, they usually start with the model.
Which model reasons better? Which one has the larger context window? Which one performs best on a benchmark?
That is becoming an incomplete way to think about agent systems.
For long-running agents, the model is only one component of the operating system around it. Planning, context management, tool design, execution controls and state management can materially change what the same underlying model is able to accomplish.
A recent empirical study of coding-agent harnesses provides useful evidence for this distinction.
Fan et al. held the core execution loop constant and independently varied three parts of the harness:
- planning;
- action space;
- context management.
The study covered four models, SWE-Bench Verified and Terminal-Bench 2.1, and 176 matched experimental settings.
The interesting result is not that one harness configuration “won”. It is almost the opposite:
That has broader implications than coding agents.
1. Context management is operational control
One of the strongest findings concerns context management.
Its value increased as the available context window became tighter. Much of the improvement did not appear to come from making the model reason better. It came from preventing execution from failing because the context window overflowed.
That distinction matters.
We tend to discuss memory and context as if they were extensions of model intelligence. In an operational agent system, they are also resource-management mechanisms.
The harness has to decide:
- What should remain in working context?
- What can be removed?
- What should be summarized?
- What evidence must remain retrievable?
The study found that relatively simple rule-based elision followed by selective LLM summarization produced a strong accuracy-to-cost trade-off.
Adding machinery for recovering previously elided content produced little benefit in the tested setting.
Every memory layer, retrieval mechanism and summarization step has operational cost and introduces another component that can fail.
The right question is therefore not, “How much memory can we give the agent?” It is, “What state does this task actually require?”
2. Planning is conditional, not universally good
Planning produced another important result.
For weaker models, explicit planning improved successful task completion. For stronger models, planning had much less effect on accuracy. Its main benefit shifted toward efficiency by reducing unnecessary work after an edit had already been made.
That suggests a broader design principle.
This matters as organizations begin replacing models underneath established agent platforms.
Suppose a workflow was originally designed around a model that needed:
- explicit decomposition;
- repeated verification;
- several specialized tools;
- aggressive memory support.
Replacing that model with a substantially stronger one does not necessarily mean the old harness remains optimal.
Some scaffolding may become redundant. And redundant scaffolding is not free.
It consumes tokens, creates latency, expands the attack surface and makes the system harder to understand.
Agent architecture therefore needs something that traditional software architecture already understands well: periodic simplification.
When capability changes, the harness should be re-evaluated.
3. More tools are not always better
The study also compared predefined tools with a much simpler action space.
Models with weaker shell capabilities benefited from specialized tools. More capable models could operate effectively with a much smaller Bash-oriented interface and, for command-line-heavy tasks, at considerably lower cost.
Again, the important lesson is conditionality.
Tooling is often presented as a capability problem: “Which additional tools can we give the agent?”
Governed architecture asks a different question: “What is the minimum action surface required for this agent to perform its role?”
This connects performance engineering directly to security engineering.
Every new tool can also represent:
- a new permission;
- another identity or credential;
- another external dependency;
- another source of untrusted input;
- another state-changing operation;
- another path requiring authorization, logging and evidence.
The minimum sufficient action space is therefore not merely an optimization technique. It is a governance property.
From static agent architecture to adaptive harness design
Taken together, these findings point toward a more mature model of agent engineering.
We should stop treating the harness as a fixed shell surrounding an interchangeable LLM.
Instead, think of the system as a combination of:
The harness can then adapt along several dimensions:
Context policy
How aggressively should history be retained, elided or summarized?
Planning depth
Does this model and task require explicit planning?
Action surface
Which tools and permissions are actually necessary?
Verification depth
How much independent checking is justified by the consequence of failure?
Governance controls
Which actions require policy checks, evidence capture or human approval?
This also changes how agent platforms should be evaluated.
Instead of asking whether “Agent A” outperforms “Agent B”, a more useful engineering question is: Which component changed the outcome, and which failure mode did it remove?
That is exactly why controlled component-level evaluation is valuable. It turns the harness from invisible orchestration into an architecture that can be measured.
The governance implication
For governed enterprise agents, I would take the idea one step further.
The harness should not merely help the model succeed. It should continuously determine the least sufficient operating environment in which the model can succeed safely.
That means avoiding both extremes.
Too little scaffolding can cause failure. Too much scaffolding creates complexity, cost and unnecessary authority.
This principle applies beyond coding agents.
A GRC agent, SOC agent, research agent or operational assistant may require very different planning, memory, tools and controls. And those requirements can change when the model changes.
The model is therefore not the entire agent. But the reverse is also true: the harness is not merely infrastructure around the model.
Together they define the effective system.
Reference
Run-Ze Fan, Zihao Zhang, Simin Ma, Yebowen Hu, Shouju Wang, Kaiqiang Song, Fei Liu, Hamed Zamani and Xiaoyang Wang, An Empirical Study of Harness Design for Coding Agents, arXiv:2609.20804, September 2026.