The short answer
The harness moves outcomes about as much as the model
Hold the model weights fixed and vary the harness or its configuration, and a coding agent's success rate can move by a few points, by twenty points, or by roughly 10x, depending on where the bottleneck sits. Through 2026 that pattern has been measured across several benchmark settings, and this article collects the measurements.
The Chinese developer community has a compact name for the pattern: thin model, thick control. The model layer is thin; the deterministic machinery around it is thick, and the experiments below measure how far that machinery can move outcomes. In English the emerging discipline is called harness engineering, and the evidence below is the case for taking it seriously.
One calibration up front, because the honest version of this thesis is stronger than the hyped one: in Claw-SWE-Bench's adversarial comparison, switching models moved scores slightly more than switching harnesses (a 29.4-point model spread versus a 27.4-point five-harness spread). The harness does not beat the model. The two spreads are of similar magnitude, yet harness choice is rarely evaluated with anything like the same rigor.
The composition
Inside Claude Code, the decision layer is thin
In a preprint submitted April 14, 2026, researchers at MBZUAI's VILA Lab dissected Claude Code v2.1.88: roughly 512,000 lines of TypeScript across about 1,884 files (arXiv:2604.14228). The paper repeats an unverified community estimate that labels 1.6% of that code as decision logic, the layer where the model chooses actions and tools, and 98.4% as the operational harness: permission gates, tool routing, context management, recovery, persistence, and execution.
Treat the split as an illustration rather than a measurement; the paper prints the figure without a counted file list or annotation rules. The architectural point survives the caveat, though, because it matches what the paper does document in detail: Claude Code's architecture is overwhelmingly deterministic infrastructure, with the model sitting inside it like a small core inside a large machine.
Composition alone does not prove the thick part matters. For that you need experiments that change the machinery while holding the model still.
The experiments
Change one harness variable, watch the score move
The cleanest single-variable result comes from Can Bölük's February 2026 edit-format experiment (the harness problem): 16 models, three file-editing formats, 180 synthetic single-file repair tasks, three runs each. With Grok Code Fast 1, the diff-style patch format scored 6.7%; his hashline format, which anchors each edit to hash-tagged line references instead of a diff, scored 68.3%. Same model, same tasks, roughly a 10x gap between two ways of writing an edit.
The result is widely misquoted, so state it precisely: the 10x is against the patch format. Against plain string replacement, the revised hashline result is about 4.6 points better. The lesson is not that one format is magic; it is that on the same tasks, the edit format alone separated 6.7% from 68.3%, and hashline beat the patch format for 14 of 16 models tested. The tasks are synthetic React repairs, so treat the sizes, not the exact numbers, as the finding.
A second experiment tests a bundled harness treatment rather than a single variable. An August 2026 preprint, Same Model, Different Harness, ran two configurations of one harness on SWE-bench under tight context budgets. With Qwen3.6 at a 20,480-token budget, the treatment bundle (trimming stale tool output, detecting stalls, injecting interventions) scored 49.1% against the baseline's 28.0% on 169 SWE-bench Verified tasks, and 32.7% against 15.2% on SWE-bench Pro. At a 262,144-token budget the advantage disappeared, 69.0% versus 68.7% on Verified, and the treatment also spent more turns and tokens, so read it as a bundle result. Within this study, the harness treatment mattered under tight budgets and not at loose ones.
The benchmarks
Across harnesses and configurations, the spread rivals a model change
The edit-format experiment isolates one mechanism; the comparisons below test whole harnesses or bundles of harness choices. Three 2026 studies, four comparisons, each with the model weights held fixed:
| Study | Fixed model | What changed | Result |
|---|---|---|---|
| Claw-SWE-Bench (arXiv:2606.12344, Jun 2026) | Qwen3.6-flash | Five agent harnesses, same prompt, workspace, and budget | 38.6% to 66.0% across harnesses, a 27.4-point spread |
| Claw-SWE-Bench, same study | GLM 5.1 | Bare adapter versus full OpenClaw-style adapter | 19.1% to 73.4%; patch-application failures fell from 69.1% to 1.5% |
| LangChain Deep Agents (Feb 2026, vendor-reported) | GPT-5.2-Codex | Middleware tuned iteratively on benchmark traces, including a reasoning-effort change | 52.8% to 66.5% on Terminal-Bench 2.0, +13.7 points |
| Self-Harness (arXiv:2606.09498, Jun 2026) | MiniMax M2.5 | Harness configuration improved by the agent itself | 40.5% to 61.9% on a 64-task Terminal-Bench 2.0 subset |
Scope notes: Claw-SWE-Bench ran each cell once, so run-to-run variance is unmeasured, and Self-Harness used its held-out tasks as regression gates during development. And the honest counterweight from the same Claw-SWE-Bench study: with the harness held fixed, switching among nine models produced a 29.4-point spread, slightly larger than the 27.4-point five-harness spread. The two variables are the same order of magnitude. Model evaluation is a mature industry habit; harness evaluation is far less standardized and far less visible.
The practice
What harness engineering asks you to do differently
Birgitta Böckeler framed the term in a February 17, 2026 memo on martinfowler.com (Harness Engineering: first thoughts), often misattributed to Fowler himself, as context engineering, architectural constraints, and garbage collection; her April 2 follow-up reframed it as designing guides and sensors around the agent. Both framings point at the same practical moves:
- Evaluate the pair, not the model. A benchmark that varies the model while silently fixing one harness measures the model inside that harness, and says nothing about how the ranking shifts under another. The measured harness spread (27.4 points) rivals the measured model spread (29.4 points).
- Look for bottlenecks first. The two most diagnostic results involve the edit format (6.7% versus 68.3% on the same tasks) and the token budget (a large bundled-treatment advantage at 20,480 tokens that disappears at 262,144). Find what is binding before tuning anything else.
- Re-measure on your own tasks. Each result lives inside its stated limits: synthetic repairs, selected SWE-bench cohorts, single-run cells, vendor-tuned Terminal-Bench runs. These effect sizes prove large differences are possible; they are not forecasts for your workload.
Where HarnessRouter fits
Treat the harness as a variable you can actually vary
Everything above assumes you can put two harnesses side by side on the same task, and that is exactly the step that is hard to take when each harness arrives as its own integration. HarnessRouter is the world's first unified interface for agent harnesses. It means one API contract for starting tasks, streaming progress, continuing sessions, and collecting files and results across every harness on the platform, so the harness becomes a request parameter rather than a rebuild.
In HarnessRouter's own recorded benchmark, eight harness and model configurations performed identical work and per-task cost varied by about 475x. Same task, same success criteria, radically different economics: the harness and model pair, chosen together, is the unit that ships.
FAQ
Frequently asked questions
Does the harness matter more than the model?
The best available comparison says they are the same order of magnitude: across one 2026 benchmark, switching harnesses moved a fixed model by up to 27.4 points while switching models under a fixed harness moved scores by 29.4 points. The practical conclusion is not that the harness wins; it is that evaluating only the model measures half the system.
What does thin model, thick harness mean?
It is a translation of a framing popular in the Chinese developer community (thin model, thick control): the model-facing decision layer of a production coding agent is small, while the deterministic machinery around it, tool routing, permissions, context management, and recovery, is large and can move outcomes materially. An MBZUAI paper on Claude Code repeats a community estimate of 1.6% decision logic versus 98.4% operational harness.
What is harness engineering?
Designing and tuning the machinery around a model, its tools and their formats, context strategy, permissions, verification, and recovery, as an engineering discipline with measurable outcomes. The term was framed by Birgitta Böckeler on martinfowler.com in February 2026, and in 2026 benchmark settings, harness changes moved fixed-model success rates by anywhere from a few points to roughly 10x, within each study's stated conditions.
Can changing a tool format really produce a 10x difference?
In one controlled experiment, yes, with scope: with Grok Code Fast 1 on 180 synthetic single-file repair tasks, a diff-style patch format scored 6.7% and a hashline format 68.3%. Against plain string replacement the gain was about 4.6 points, so the finding is that the tested formats produced sharply different results on the same tasks, not that one format is universally 10x better.
Pick the best harness and model for every task
Run real tasks across harnesses through one API, compare success, latency, and recorded cost, and route each task class to the measured winner.
Start building free