The Harness Is the Product: Why the Wrapper Around Your Coding Agent Now Outweighs the Model You Picked

Why Do Two Teams Running the Same Model Get Different Results?

Most engineering organisations still treat agent performance as a procurement question. Somebody reads a benchmark table, picks the model at the top of it, signs the contract, and expects the results in the table. When the results do not arrive, the conclusion is usually that the model is overhyped, or that the codebase is too complicated, or that agents are simply not ready yet. All three can be true. None of them is usually the reason.

The cleanest demonstration of what is actually going on comes from security tooling rather than from the model labs. Endor Labs ran an identical model through two different agent harnesses across 200 real vulnerability-fixing tasks embedded in working codebases, with no security labels on the tasks and nothing in the instructions beyond a general direction to follow security best practices. Under one harness the model produced a patch that passed the functional tests 59.8% of the time. Under the other, the same model, the same tasks, the same week: 72.6%. The security pass rate, which additionally requires the patch to survive hidden tests derived from the original fix, moved from 19.0% to 29%.

That is a 12.8 point swing in functional correctness and a 10 point swing in security, produced by changing nothing that appears on a pricing page. The gap is larger than the gap between many adjacent model releases, and it was not caused by one harness having more time or more compute. Endor's analysis put it down to patch quality: one harness steered the model toward validating inputs across every data-flow sink, the other accepted a fix that handled the obvious path and stopped.

The same effect shows up when a team optimises deliberately rather than by accident. LangChain published a detailed account of moving its own coding agent from 52.8 to 66.5 on the 89 tasks of Terminal Bench 2.0, going from roughly thirtieth place to roughly fifth, on a fixed model. They changed three things: the system prompt, the tools, and the middleware hooks wrapped around model and tool calls. The model stayed exactly where it was.

Once you have seen two results like that, the strategic question changes shape. It stops being which model is smartest and becomes which system makes a given model's intelligence reliably useful on your code. That second question is an engineering problem with an owner, a budget and a test suite, which is considerably better news than the first one.

What Is Actually Inside an Agent Harness?

The vocabulary is still settling, which is part of why the layer gets overlooked. Hugging Face's agent glossary draws the line usefully: the harness is the execution layer that calls the model, handles its tool calls and decides when to stop, while the scaffold is the behavioural frame around it, the system prompts, tool descriptions, response parsing and context management that shape what the model perceives at each step. In practice most teams use harness to mean both, and an agent is the model plus all of it.

Faros describes a production harness as five layers, and the decomposition is worth borrowing because each layer has a different owner and a different failure signature. Tool orchestration governs how the agent selects and chains tools and how it recovers when one fails. Verification loops run automated checks during execution so that a wrong turn fails fast instead of compounding for forty more steps. Context and memory cover codebase indexing and what persists across turns and sessions. Guardrails are the hard boundaries: budget ceilings, forbidden operations, human approval gates. Observability is the telemetry and audit trail that makes any of the other four debuggable.

None of those five layers is exotic. Every one of them is ordinary software engineering, the kind a competent backend team has been doing for twenty years under different names: retry policy, circuit breaker, cache, authorisation, tracing. That is the encouraging part. The discouraging part is that they are ordinary software engineering, which means they have to be built, tested, owned and maintained by someone, and they do not arrive with the model.

The layer that most teams underbuild is verification, and it is the one with the highest leverage. An agent that writes code and stops has produced a suggestion. An agent that writes code, runs the build, reads the failure, and iterates has produced something closer to work. LangChain's account is explicit that the harness has to inject the environmental context deterministically for the agent to fall into that build-verify cycle reliably, rather than hoping the model chooses to test its own output.

The layer that most teams overbuild is the system prompt. It is the easiest thing to edit, so it absorbs effort that would pay back far better in tools and middleware, and it grows into a two-thousand-word document of accumulated superstition that nobody dares delete a line from.

Why Do So Few Teams Have Any Harness Discipline?

There is now hard data on how little of this layer actually exists in the wild. An empirical study of 2,853 GitHub repositories by Galster, Treude, Baltes and colleagues catalogued eight distinct configuration mechanisms across Claude Code, GitHub Copilot, Cursor, Gemini and Codex, then measured adoption. Context files dominate, and are frequently the only mechanism present in a repository at all, with AGENTS.md emerging as the interoperable standard across tools. The advanced mechanisms, skills and subagents, are adopted by very few repositories, and the skills that do exist rely predominantly on static instructions rather than executable scripts.

Read that as a maturity snapshot and it says something uncomfortable: the median team's entire harness is one markdown file. The loop is whatever the vendor shipped, the tool surface is whatever the vendor shipped, the verification is whatever the model decides to do that day, and the observability is the scroll-back buffer in a terminal. That is not a criticism of anyone's engineering judgment. It is what happens when a capability arrives as a product rather than as a library, and the product works well enough in a demo that nobody budgets for the layer underneath.

The cost of that gap surfaces later, at the point where a pilot is supposed to become a system. Gartner predicts that over 40% of agentic AI projects will be cancelled by the end of 2027, attributing the failures to escalating costs, unclear business value and inadequate risk controls. Every one of those three is a harness property. Cost is set by context management. Risk control is guardrails. Demonstrable value requires observability good enough to attribute an outcome to an agent in the first place.

There is a second reason the layer stays thin, and it is organisational rather than technical. Harness work has no natural home on most org charts. It is not platform engineering, because it touches prompts and model behaviour. It is not ML, because there is no training involved. It is not application engineering, because it is infrastructure. So it gets done in the margins by whoever is most enthusiastic, which means it is undocumented, unowned, and it leaves when they do. StepTo has made a closely related argument about institutional knowledge and skill files, and the pathology is the same one in a different coat.

The Failure Modes That Only Show Up in Traces

Harness defects do not look like bugs. They look like an agent that is sort of working, most of the time, in a way that is hard to argue with in a standup. Finding them requires reading traces, which is why observability sits in the five layers rather than being an optional extra.

The most instructive documented example is the doom loop. LangChain's trace analysis found agents becoming myopic once they had settled on a plan, then making small variations on the same broken approach, more than ten times in some traces, while burning tokens and wall-clock the entire way. No exception is raised, no test goes red, and the session eventually ends with something plausible. Their fix was not a better prompt but a middleware hook, a loop detector watching for repeated edits to the same file that interrupts the agent and forces it to reconsider direction.

The second failure mode is configuration that looks like an upgrade and is not. LangChain's own published numbers make the point sharply: pushing the reasoning budget to its highest setting scored 53.9%, below the 63.6% achieved at the setting one notch down, because the extra deliberation pushed tasks into timeouts. More thinking, worse outcomes, on the same model and the same harness. Every knob in a harness is like this. None of them is monotonic, and the only way to know which direction yours should go is to run the tasks.

That is the deeper point about harness work: it is empirical, not architectural. You cannot reason your way to the right tool granularity or the right compaction threshold from first principles, and anyone who claims otherwise is selling something. Academic work is already formalising the loop, with observability-driven automatic evolution of coding-agent harnesses using execution traces to identify ineffective components and evolve the scaffolding around them. Whether or not you automate it, the shape of the work is the same: instrument, run a fixed task set, read the failures, change one thing.

Which means harness engineering has a hard prerequisite that most teams have not met. You need a task set that represents your actual codebase and produces a number. Without it you are tuning by vibes, and vibes are precisely what the harness exists to replace. This is the same discipline we argued for in our piece on the evaluation gap, and harness work is where the absence of evals stops being a governance concern and starts being a daily obstacle.

How the Harness Decides Your Token Bill

The financial argument for harness engineering is more concrete than the quality argument, and for a CFO it is usually the more persuasive of the two. The cost of an agentic session is not driven by the headline price per million tokens. It is driven by how many times your harness resends the same context.

Vantage's analysis of agentic coding spend puts the mechanism plainly: input tokens outnumber output tokens by 20 to 25 times in these workflows, because every turn resends the system prompt, the retrieved files and the accumulated conversation. A session that begins at roughly 5,000 input tokens per turn can be sending 35,000 or more by turn fifty. The agent is not doing seven times more work at that point. It is re-reading its own history seven times over, and paying for it each time.

The absolute numbers are large enough to matter at team scale. Vantage models a fifty-turn session at around a million input tokens and forty thousand output tokens, which lands at roughly $6.00 on the most expensive frontier configuration and roughly $0.60 on a cheaper one, a tenfold difference on the same volume of work. Extended to a twenty-five person team running about a thousand sessions a month, that is the difference between approximately $72,000 and approximately $7,200 a year.

The lever that closes most of that gap is context management, which is a harness concern and nothing else. Compaction strategy, whether history is summarised or snipped or spilled to files, whether subagents get isolated context windows instead of inheriting the parent's, whether tool results are truncated before they enter the transcript: these choices set your bill far more decisively than the vendor you chose. Our earlier treatment of AI cost engineering covers the FinOps mechanics; the harness is where the mechanics get implemented.

There is a corollary that engineering leaders should take seriously when negotiating budgets. Faros tested open-weight models against frontier models across 211 real engineering tasks and found the optimised open models matching or outperforming the expensive proprietary alternatives when paired with a robust harness. If that holds on your workload, harness investment is not merely a quality play. It is the thing that converts a model licence from a fixed cost into a negotiable one.

Why the Harness Is a Security Boundary

The Endor Labs result deserves a second look, because the security half of it is the part with regulatory consequences. The same model, on the same 200 vulnerability-fixing tasks, produced patches that passed hidden security tests 19.0% of the time under one harness and 29% under another. Both numbers are low in absolute terms, which is its own warning. The gap between them is the part that is actionable.

What that gap represents is the harness deciding how thoroughly the model looks. A harness that surfaces the full data-flow around a change, that runs the test suite before declaring victory, that requires the agent to state what it checked, produces materially safer patches from identical weights. A harness that treats the first plausible diff as the answer does not. Nothing in your model contract governs this. It is entirely a property of code you either wrote or accepted by default.

The guardrail layer carries the rest of the security surface, and it is where the harness stops being a performance concern and becomes a control. Which tools can the agent call without approval. What is the spend ceiling on a single session. Can it write outside the working tree, reach the network, read secrets from the environment, push a branch. Those are authorisation decisions, and treating them as configuration defaults is how organisations end up with the sprawl and over-privilege problems StepTo has written about in agent registries and lifecycle governance and in execution sandboxing and isolation.

For European teams there is a compliance dimension that makes the harness the natural place to implement several obligations at once. Human approval gates, audit logs of agent inputs and outputs, and demonstrable boundaries on autonomous action are exactly the artefacts that documentation obligations demand, and the harness is the only layer where all three can be enforced uniformly across whatever models you happen to be using this quarter. Build them once in the wrapper and model swaps become a configuration change rather than a re-certification.

Should You Build, Buy, or Configure?

The honest answer for most organisations is configure, with a small amount of build, and almost never a full in-house harness from scratch. But the decision deserves more than a default, because the three options fail in different ways.

Buying means adopting a vendor harness and accepting its opinions. This is right for the majority of general coding work, and the benchmark numbers above support it: the vendor harnesses are where the most tuning effort has gone. The cost is that its opinions are not tuned to your repository, its context strategy sets your bill, and its guardrails are set for a generic customer. You are also, as the Endor comparison shows, making a consequential quality decision when you pick one, which means it deserves an evaluation rather than a preference.

Building the whole thing is right for a narrow set of cases: agents operating on proprietary internal systems with no vendor tool coverage, workloads where per-session economics dominate at scale, or regulated environments where the audit trail has to be yours. It is expensive, it is ongoing, and the maintenance burden is real because the underlying models keep moving.

Configuring is the option most teams skip and most teams need. It means taking a vendor harness and doing the work the study of 2,853 repositories found almost nobody doing: real tool definitions for your build and test systems, subagents with isolated context for the tasks that pollute a transcript, middleware that detects your failure patterns, a compaction policy chosen deliberately, and a fixed task set that tells you whether any of it helped. This is a few engineer-weeks, not a programme, and it is where the LangChain-shaped gains live.

Wherever you land, sequence it the same way. Build the eval set first, instrument second, change one variable at a time third. Teams that reverse this order end up with an elaborate harness, a strong opinion about it, and no evidence.

Who Actually Builds This, and Why It Is Hard to Hire For

Harness engineering sits in an awkward gap in the labour market. It is not a research role, so ML specialists are both overqualified and pointed at the wrong problem. It is not a frontend or product role. The people who are genuinely good at it look like senior backend or platform engineers who have spent real time reading agent traces, which is a combination the market has barely started pricing, and which almost nobody can hire for quickly on a permanent requisition.

It is also, awkwardly, work that senior people have to do. Tool granularity, compaction thresholds and guardrail boundaries are judgment calls about your system's failure modes, and the person making them needs to already understand what breaks in your architecture. Handing harness work to a junior because it looks like configuration is a reliable way to end up with a two-thousand-word system prompt and no verification loop.

This is the shape of engagement StepTo is built around. We are a senior-led nearshore partner based in Serbia, working with European and US clients in overlapping hours, and the overlap matters more on this work than on most. Harness tuning is an iterative loop of run the task set, read the traces, change one thing, and that loop runs at the speed of your slowest feedback cycle. It is a good day's work across a two-hour timezone gap and a two-week round trip across a twelve-hour one, which is the practical reason harness quality tends to be poor in far-offshore arrangements regardless of individual skill.

The capability split is fairly clean in practice. The loop, the middleware, the tool layer and the eval harness are AI engineering work. The guardrails, sandboxing and permission model are security engineering and should be scoped separately, because they have a different acceptance test and a different reviewer. The upstream question of which workflows should be handed to agents at all, rather than how to wrap them, is AI strategy work, and separating it from the build keeps it from being decided by whoever happens to be editing the prompt that week.

For the build itself we default to a dedicated development team rather than fixed-scope delivery, because a harness is a system you operate rather than a project you finish: models change, your codebase changes, and last quarter's compaction policy quietly stops being right. Where the gap is narrower and the knowledge is already yours, staff augmentation puts senior engineers inside your existing team, which is often the better shape when what is missing is capacity rather than direction. Our rates are on the pricing page, how we run engagements is on the process page, and the case for nearshore development specifically is set out in our Serbia overview.