Benchmark 01 · Care Prep
Same task. 8 configurations. ≈475× cost range.
One controlled task on Codex, Claude Code, and Hermes across four models. Cost ran from 0.47 to 223 credits; strict grounding from 0% to 67%.
One controlled test, not a universal ranking.
Pareto frontier: nothing cheaper and at least as groundedNot plotted: Hermes · claude-sonnet-4.6, no completed runs
| Harness | Model | Credits | Latency |
|---|---|---|---|
 Hermesgpt-5.2 | gpt-5.2 | 0.47 | 2m 33s |
 Codexgpt-5.2 | gpt-5.2 | 0.72 | 4m 36s |
| Hermesclaude-sonnet-4.6 | claude-sonnet-4.6 | 39.6 | 2m 49s |
| Hermesgpt-5.5 | gpt-5.5 | 40.7 | 1m 25s |
| Codexgpt-5.5 | gpt-5.5 | 59.7 | 2m 46s |
 Claude Codeclaude-sonnet-4.6 | claude-sonnet-4.6 | 83.5 | 1m 31s |
| Hermesclaude-opus-4.8 | claude-opus-4.8 | 150 | 2m 42s |
| Claude Codeclaude-opus-4.8 | claude-opus-4.8 | 223 | 3m 18s |
01Findings
The model sets the range. The harness moves it.
- 1.5× to 2.1×
- The harness alone. With the model held constant, switching only the harness moved cost by 1.5× to 2.1× and latency by up to 1.95×.
- ≈475×
- Model tier, then harness. Most of the 475× spread comes from model-tier economics; on the same model the harness multiplies cost by up to 2.1×. Both choices are parameters you can route on.
- 6 of 8
- Dominated configurations. For 6 of the 8 configurations, another setup in this test was both cheaper and faster.
02Method
How to read it.
- Test scope
- One controlled synthetic-data Care Prep experiment across 8 harness and model configurations.
- Observed history
- From the same controlled synthetic-data experiment, which retained 5 runs per configuration and applied an objective validator to completed outputs.
- Held constant
- The dataset, task, Skill, schema, and output contract remained fixed across configurations.
- Changed
- The selected agent harness and model configuration changed between rows.
- Measurements
- The source capture records cost and end-to-end latency for each configuration. The observed history reports execution success, strict grounding pass rate, and p95 duration.
- Quality boundary
- Strict grounding pass checks schema compliance, exact supplied facts, evidence-reference validity, and required human-review flag. It is not a composite quality score or clinical validation. This publication does not combine quality into one score; safety and workflow usefulness remain separate human-review criteria.
- Interpretation
- These results apply to this task and configuration set. Results can change with the task, input, harness, model, environment, and configuration.
- Source
- The table transcribes two recorded captures: cost and latency and grounding. Published 2026-08-10. We will publish more tasks, configurations, and evaluation methods over time.
03Your turn
Your workload is the benchmark. Run yours.
Compare harnesses on the work your product actually runs. HarnessRouter Cloud records execution traces across harness and model configurations, so you can compare cost, latency, and quality by your own production criteria.