Design guide
Computer-use agent evaluation.
Computer-use agent evaluation checks what an agent accomplished under a documented task, environment and budget. Self-reported completion, verified task success and human preference measure different things. Report evaluator coverage, failures and uncertainty alongside outcomes on held-out tasks.
What should an agent evaluation measure?
Begin with the capability you want to test: selecting the right record, completing a cross-application workflow, recovering from an error or coordinating with another agent. Define the task and acceptable result before running the model. Preserve the environment version, agent configuration and allowed tools with each attempt.
A useful evaluation can explain both the result and its limits. An agent that says “done,” a user who prefers its answer and a verifier that accepts its final artifact provide different evidence. Combining them into one undefined success number hides information a researcher needs.
Separate completion, verified success and preference
| Signal | What it establishes | What it can miss |
|---|---|---|
| Agent-reported completion | The agent declared its run finished. | Missing requirements, incorrect records or unintended changes. |
| Verified task success | The available outcome checks accepted the recorded result. | Requirements or side effects outside those checks. |
| Human preference | A judge preferred one attempt under the stated comparison. | Hidden state, factual errors or criteria the judge could not inspect. |
CoArena’s methodology makes these distinctions explicit. Its completion metric records agent-reported finishing, while independent checks support verified completion only where an executable evaluator is available. Apply the same care to your own metric names and denominators.
Build a verifier around the result
A task verifier checks evidence against acceptance criteria. Depending on the task, that evidence may be an application record, a generated file, a rendered artifact or a combination. OSWorld illustrates execution-based evaluation tied to task-specific setup and result checks.
For a hypothetical invoice task, checking that a PDF exists is insufficient. Check that its customer, line items and total agree with the requested invoice, that the application saved the corresponding record and that unrelated invoices remain unchanged. Keep privileged reference artifacts and verifier internals outside the agent’s observation channel, while supplying every task requirement the agent is meant to know.
Test the verifier itself with correct results, partial results, misleading filenames and plausible but wrong records. Record an indeterminate result when the evidence cannot be inspected. Do not silently turn an evaluator outage into an agent failure or discard it from reporting.
Use comparable budgets and precise endings
Report both action and elapsed-time limits where relevant, together with retry policy, model settings and tools. Record time spent waiting on external applications. Different budgets answer different questions; a result from a short run should not be presented as directly comparable to an unrestricted attempt.
Distinguish a task-defined terminal state from an external cutoff. Gymnasium’s treatment of termination and truncation explains why this matters for learning. In an evaluation report, additionally label infrastructure interruption, user cancellation and policy refusal. Each ending changes how a result can be interpreted.
Keep held-out tasks genuinely separate
Reserve evaluation tasks before tuning prompts, rewards or the agent harness. Separate task families and source material where possible: renaming a customer in an otherwise identical workflow may test parameter variation without testing a new skill.
Track which tasks informed development, restrict access to held-out answers and retain versioned manifests. Check for copied templates and related artifacts across splits. These practices reduce leakage risk; they do not prove that a frontier model has never encountered similar public material. Describe what you controlled instead of claiming contamination-free evaluation without evidence.
Report failures and evaluator coverage
Publish the number of scheduled tasks, started runs, valid attempts and outcomes the evaluator could decide. Explain exclusions and report success within the verified subset alongside its coverage. A high rate on a small, easy-to-grade subset does not establish equivalent performance on the full workload.
Retain a failure taxonomy that separates wrong actions, stale observations, unmet prerequisites, application errors and grader errors. Use traces to explain examples, and repeat trials when behavior is variable. Report uncertainty and task mix rather than treating small differences as a reliable ranking.