October 7, 2026
By
Joshua Goldfein
An agent pilot that misses usually produces a proposal for a bigger model. The cheaper diagnosis is the layer around the model, and one public experiment shows how far that layer alone can move the score.
In short
The meeting has a shape. The pilot demo was good, the production numbers are not, and the deck on the table proposes the newest model. Nobody in the room can say what failed, so a bigger model feels like the responsible answer.
It is the wrong first move, because the model is the one component you cannot read. Everything around it you can: the tools the agent calls, the context it is handed, the permissions it holds, the checks on its output, and the written definition of done. Engineers call that layer the harness. Its failures show up in a log, get fixed in days, and can be measured on the next run, all before anyone signs for upgrade spend.
What the log says
Pull the traces from a disappointing agent run and the model is rarely the first thing you find. Three failures are worth looking for before any other diagnosis.
01
Wrong place
The agent finished the work and wrote it somewhere nobody reads.
02
Wrong shape
It produced a usable analysis in a format the next system could not parse.
03
False done
It hit a permissions wall, guessed, and reported success, because nothing had defined what success meant.
Those are workflow failures. A smarter model inherits every one of them, with better grammar.
The harness, in components
The harness is everything that turns a model into a worker with a job. Five components cover most of it.
| Component | What it covers | When it is missing |
|---|---|---|
| Tools | What the agent can call, and what happens when a call fails | A silent tool failure becomes a confident wrong answer |
| Context | The brief, the relevant history, and the sources it is allowed to use | The agent works from whatever it finds first |
| Permissions | Which systems it can enter, and which actions need a sign-off | It acts alone on the one decision that needed a human |
| Output contract | The format, destination, and validation of what it produces | A deliverable lands in the wrong place and does not count |
| Definition of done | The written completion criteria | The agent decides for itself when the work is finished |
In plainer terms, the model is a talented new hire and the harness is the job. A modest model in a strong harness ships predictable, reviewable work. A frontier model in a weak harness ships impressive fragments that never land anywhere.
One public experiment, with its receipts
Joel Niklaus put a number on this in “Don’t Train the Model, Evolve the Harness”, published on Hugging Face on July 1, 2026. The setup is the clean version of the argument: freeze one open model, DeepSeek-V4-Pro, with no fine-tuning and no swap, then let an automated loop rewrite only the scaffold around it, scored against Harvey’s Legal Agent Benchmark with the same judge throughout.
SAME MODEL, SAME JUDGE. ONLY THE HARNESS CHANGED.
Those are held-out test figures: the share of graded criteria met, and the share of tasks with everything right end to end. On the development set the loop optimized against, criterion pass went from 63.1% to 83.3%. Across every wrapper the loop tried, pooled pass rates ran from 3.5% to 80.1% with the model and the judge held constant.
Two details matter more than the headline. The article reports a 6.4-point move on the dev set from a single deliverable landing gate, a deterministic check that the finished file exists where the task expects it. And five of the top six harnesses were deterministic code: the article says code and robustness mechanisms transferred across model families, while prompt playbooks were model-specific and could backfire.
The caveats come from the same source. Whole-task success went from 0% to 5.0%. Better, and still low. The search hit what the article calls a local ceiling, and the remaining failures, in the author’s words, likely need some mix of better prompts, better tools, and a stronger base model. Model capability sets the ceiling. The harness decides how close you get to it.
What this licenses
Anyone claiming the harness always beats the model is selling something. The defensible claim is about sequence. Harness work is legible: read the log, fix the landing gate, rerun, measure. A model upgrade without that diagnosis is a bet placed blind. Make the legible fix first, measure what is left, then price the upgrade against the failures that remain. Often you will want both, in that order.
Six questions before you sign the upgrade
-
Is “done” written down?
If completion criteria live in someone’s head, the agent grades its own homework.
-
Where do deliverables land, and does anything check they arrived?
The landing gate above was worth 6.4 points on its own.
-
What tools does the agent have, and what happens when one fails?
Silent tool failures become confident wrong answers.
-
Does anything validate output before a human sees it?
Format checks, required fields, sanity bounds: deterministic, cheap, and run on every output.
-
What are the permission boundaries and approval gates?
An agent that can act without sign-off will eventually act wrongly without sign-off.
-
Can you replay a failure?
Without traces, every diagnosis is a guess and every fix is a hope.
If you cannot answer all six, a new model changes little beyond the invoice. Answer them, fix what they expose, and rerun the workflow before anyone prices the upgrade.
Anyone claiming the harness always beats the model is selling something.
Boundary condition
Where Mercury fits
A bounded agent reliability audit does exactly that work on one workflow. We instrument the failures, separate harness problems from capability limits, and size each before anyone approves upgrade spend. Where the diagnosis points at the harness, we build it: a written definition of done, delivery contracts and output validators, approval gates on the actions that carry risk, and evals that catch the regression before the next model swap does. If you have an agent that disappoints and no clear account of why, that audit is the first useful step.
The experiment is one author’s self-reported result on one legal benchmark with one model. Mercury has not reproduced it, and it says nothing about your workflow. What transfers is the method, not the numbers.
