
Eval harness runs on every model change
Teams change prompts and models often and check the results by trying a few examples. An agent runs the whole evaluation suite on every change, compares it with the last approved run and writes the report. Whether the change ships is a person’s call.
The short answer
An agent runs your evaluation suite on every change by watching for a new model version, prompt, retrieval setting or tool definition, running the full set of evaluations you defined, and comparing the results with the last approved run.
It reports what improved, what regressed and which cases failed. A named person decides whether the change ships.
Why do model changes ship after a few spot checks?
Someone tweaks a prompt to fix one bad answer, tries three examples, and ships it. Next week a model upgrade arrives, and the same thing happens. Nobody reruns the cases that worked before, so a fix in one place quietly breaks another, and the first person to notice is a customer. The evaluation set exists, but running it is slow and manual, so it runs before big launches and not much else.
Who is this for?
An ML engineer, AI product lead or engineering manager whose product depends on a model, prompts or retrieval settings that change often. It suits teams with at least a starting evaluation set: test inputs with expected outputs, or criteria for judging them. Deciding what to measure, and what is good enough to ship, stays with the team.
What changes when the full suite runs every time?
| Part of the release | Spot-checked by hand | Evaluated on every change |
|---|---|---|
| What is tested | A few examples someone remembers | The whole evaluation suite you defined |
| When | Before big launches | Every model, prompt or setting change |
| Comparison | Against memory | Against the last approved run, case by case |
| Regressions | Found by customers | Listed in the report before release |
| The ship decision | The person who made the change | A named approver, with the report in hand |
How it works, step by step
The agent runs and compares. The team defines the measures and decides what ships.
Define the suite with the team
Watch for changes
Run the full suite
Compare with the last approved run
Send rubric cases to reviewers
Human gate: a named approver decides whether the change ships
Record the decision and the new baseline
What does the agent never do with a model change?
Ship it
Deployment follows the approver's decision, through your normal release.
Score judgment cases. Rubric cases go to people.
Move the baseline
Only an approved run becomes the new baseline.
Test on live users
Runs happen in a test environment.
Pick the metrics. The team decides what is measured. Google's Rules of Machine Learning put it early in the list: first, design and implement metrics.
Where this fits
One change taken from detection to an approval decision is a single process: Tier 1, Automate a Workflow. The readiness checklist will look at whether your evaluation set is written down well enough to run. The same practice of comparing new output with an approved baseline before trusting it runs through shadow mode, and the approval follows approval gates. More in the use case library.
How do model evaluations differ from unit tests?
| Question | Unit tests | Model evaluations |
|---|---|---|
| What is checked | Code behaviour | Model output quality |
| Pass or fail | Usually exact | Often graded, sometimes by people |
| What triggers a run | A code change | A model, prompt, setting or data change |
| Who decides on a regression | The build fails | A named approver decides |
Related reading
- Shadow mode: comparing new output with a trusted baseline before relying on it.
- Approval gates: why the ship decision has one name.
- The governed research brief: the same habit of checking output against sources.
