Quick answer
What a model eval is
A model eval is a repeatable test that measures how candidate models perform on your application's real tasks, so model choice comes from your own data instead of a leaderboard or a recommendation.
The method is simple to state: collect representative tasks from your product, define what a good result looks like, run every candidate model against those tasks in parallel under identical conditions, and compare on success and quality first, then cost and latency among the candidates that pass. Rerun the same suite whenever a model updates, a provider deprecates a version, or your product changes what it asks for.
Most teams skip this and choose models by reputation. This page covers how to build an eval that actually predicts production behavior, and then the part most eval guides stop short of: what your model eval holds constant, and why that constant deserves an eval of its own.
The problem
Why leaderboards keep picking the wrong model
Public benchmarks measure general capability across broad task sets. Your product rewards one narrow behavior: extracting the right fields from your documents, calling the right function in your API, writing in your voice for your users. A model can sit at the top of a leaderboard and still lose on your workload to a model two tiers cheaper, because the leaderboard never asked your question.
Reputation-driven selection has a second failure mode: it is static. Models update, prices change, and new releases arrive continuously, but a choice made on reputation gets revisited only when something breaks. An eval turns model choice from a one-time bet into a measurement you can rerun on every release.
Method
How to build a model eval for your use case
A useful eval has four parts: fixtures, graders, a fair harness for running candidates, and a schedule.
- Fixtures: pull real tasks and inputs from your product, including the messy ones that caused support tickets. Synthetic prompts test a model's manners; production fixtures test its job performance.
- Graders: use exact checks wherever the output is checkable (did the code run, did the extraction match, did the right function get called with the right arguments). For open-ended output, use LLM-as-a-judge with a written rubric, and spot-audit the judge against human ratings so you know how far to trust it.
- Agent tasks need behavior grading, not just answer grading: score which tools the model called, which tools it correctly avoided, and whether the task actually reached a finished state.
- Fairness: pin everything except the model. Same fixtures, same prompts, same tool surface, same environment. Run each candidate multiple times, because single runs of nondeterministic systems prove nothing, and report the distribution, not the best run.
- Schedule: wire the suite into CI so a model update, a provider deprecation, or a prompt change triggers a rerun. An eval you run once is a snapshot; an eval on a schedule is a regression net.
Apply a success threshold before ranking anything on price: a cheap run that failed is not a bargain. Among candidates that pass, cost and latency become the tiebreakers, and the winner is often not the model you would have guessed.
Read the fine print
The variable every model eval pins
Notice what the fairness rule did: to make model comparisons clean, it froze the execution layer. Same prompts, same tools, same environment, only the model changes. That is correct method, and it also defines the scope of the answer. A model eval tells you the best model for your current execution layer. It says nothing about the layer itself. That layer is the harness: the system that manages context, drives the tools, and carries the task to completion. Whether the harness is the right one for your workload is a question the eval never asks.
For a single prompted completion, there is little execution layer to speak of. For agent workloads, where the model plans, calls tools, and iterates toward a result, the harness is a second axis with measurable weight of its own. How much weight is a question recorded data can answer.
Recorded evidence
What varying the pinned layer does
In HarnessRouter's same-task benchmark, eight harness and model configurations ran identical work with identical input, five runs per configuration. Across configurations, cost varied by about 475x, and the lowest-cost successful run used 99.8% fewer credits than the costliest configuration. Success rates and scores differed by setup, and the costliest configuration was not the best one.
The finding is scoped to that recorded task, and specific winners vary by workload. Where did the spread come from? Mostly from model-tier differences. But the harness multiplied on top of that: with the model held constant, swapping only the harness moved cost by up to about 2.1x and latency by up to about 1.9x. Success rates differed by setup too. The spread is a property of the combination: not of the model alone, and not of the harness alone.
Put next to a model-only eval, the conclusion is straightforward. The model is the biggest lever, and a good model eval pulls it well. The harness is a second lever that a pinned-harness eval cannot see. In the recorded runs, that second lever was worth up to about 2.1x on cost. In some setups it decided something bigger: whether the task succeeded at all. When trying another configuration is a parameter change rather than a rebuild, there is little reason to leave it unmeasured.
Side by side
Standard model evals vs evals on HarnessRouter
The two are not rivals; they answer at different depths. A standard model eval stack, whether hand-built or from an eval tool, is the right instrument when your application makes direct model calls. Evals run on HarnessRouter operate one level up: the same task goes to complete harness and model configurations, and the comparison covers everything your users actually experience.
| Capability | Standard model evals | Evals on HarnessRouter |
|---|---|---|
| Rank candidate models on your own fixtures | Core capability | Core capability |
| Compare harness × model configurations on the same task | Not this layer's primary job | Core capability |
| Cost and latency measured over complete agent tasks, not single calls | Varies | Core capability |
| Full execution traces behind each recorded result | Varies | Core capability |
| Adopt the winning configuration with a parameter change | Not this layer's primary job | Core capability |
The rows are a scope statement, not a scorecard: a model eval cannot see the configuration axis, because pinning that axis is exactly what makes it a clean model eval. If your product ships agent features, run the eval at the level you ship.
One level up
How to run a configuration eval
The method mirrors a good model eval, one level up. Fix the task and the input. Run every candidate configuration against it multiple times. Apply the success threshold first, then compare cost and latency only among configurations that met it. Record everything so the comparison is auditable rather than anecdotal.
The practical obstacle has never been the method; it is that each harness normally requires its own integration, so comparing configurations meant building the same plumbing several times. That is the layer HarnessRouter exists to remove. HarnessRouter is the world's first unified interface for agent harnesses: one API through which your product runs complete agent tasks on Codex, Claude Code, or Hermes, with sessions, streaming, files, and results handled the same way across all of them. Because every harness runs behind the same contract, a configuration eval becomes a parameter sweep, and execution tracing lets you observe runs, compare configurations, and pick the best one on evidence.
FAQ
Model eval FAQs
What is a model eval?
A model eval is a repeatable test that measures how candidate models perform on your application's real tasks. It uses fixtures drawn from production, graders that check success and quality, and identical run conditions for every candidate, so model choice comes from measured results instead of leaderboards or reputation.
How many runs does a model eval need?
More than one per candidate. Model and agent runs are nondeterministic, so a single run can flatter or slander a candidate. Run each candidate several times per fixture and compare distributions. HarnessRouter's published benchmark uses five observed runs per configuration for this reason.
Do I need eval expertise to get started?
No. Start with a dozen real tasks from your product, exact checks for whatever is objectively checkable, and a success threshold. Eval tooling can scaffold test files and graders for you, but the parts that make an eval predictive, production fixtures and an honest definition of success, come from knowing your own product, not from eval expertise.
What is LLM-as-a-judge and when should I trust it?
LLM-as-a-judge uses a model with a written rubric to grade open-ended outputs that exact checks cannot score. It scales well but inherits model biases, so spot-audit its verdicts against human ratings before relying on it, and prefer exact checks wherever the output is objectively checkable.
Do agent workloads need more than a model eval?
Yes. A model eval pins the execution layer, but for agent tasks the harness and model configuration determines cost, latency, and success together. In HarnessRouter's same-task benchmark, cost across configurations varied by about 475x, with harness choice alone moving cost by up to about 2.1x on the same model. Evaluate the configuration your users actually experience.
Pick the best harness × model for every task
Run the same task across configurations through one API, compare success, latency, and cost, and ship the winner.
Start building free