Choose harness and model combinations from traces.

Principle

No single harness or model setup wins every task. Treat each configuration as a hypothesis until your own product tasks produce evidence.

Compare dashboard Trace, success, elapsed time, stream behavior, recovery, files, and artifact quality before promoting a configuration into production.

What to compare

Task scenario

Use the end-user job as the unit of comparison, such as contract review, codebase repair, document generation, or an internal workflow action.

Harness

Compare configured agents by their runtime purpose, instructions, tools, skills, permissions, request headers, and artifact contract.

Model

Fetch the current model catalog and allowed models server-side. Do not hardcode a permanent model default or assume token count alone explains agent quality.

Policy

Include runtime limits, data access, sandbox behavior, approval boundaries, request headers, and whether the task used production data or a non-production fixture.

Decision loop

• Define the artifact your UI needs to render or download.
• Create one or two candidate configured agents with only the permissions and tools the task needs.
• Run representative tasks through the same authenticated product server route your users will use.
• Save response.id and response.metadata.session_id from response.created, then inspect dashboard Trace, stream events, session status, turns, files, elapsed time, and the final artifact.
• Promote the best candidate only after the product path repeatedly returns the expected artifact, handles recovery, and keeps file preview/download ownership checks intact.

When to split configured agents

Use a separate configured agent when traces or product requirements show a different purpose, data boundary, tool set, permission model, model policy, risk boundary, or artifact contract. Do not split by button count alone.

Decision record

• Record the selected feature_key, harness_id, default model policy, and artifact contract.
• Store why this combination won: completion behavior, elapsed time, recovery behavior, file output, and dashboard Trace observations.
• Keep rejected candidate notes short enough to reproduce the comparison later.
• Update the server-side feature mapping only after the chosen configuration passes through the product UI.