Aviral GargOne agent leaves five tasks unfinished; another leaves two. The second costs more in total. The lower bill leaves more work undone. Comparing them starts with deciding what a finished task is worth.
Choose a concrete result before running a comparison. For a coding task, that might mean a patch passes the agreed checks and a reviewer accepts the change. A confident explanation alone does not establish that result.
Anthropic’s evaluation guide separates the agent’s actions from the resulting environment state. For a patch, read the changed code and test result; the agent’s success message is only a report about the work.
Use the same task set, acceptance rule, tool access, harness version, time budget and retry policy. A harness is the setup that runs and checks the work. Record the exact model version and configuration too.
SWE-bench’s evaluation guide offers a concrete example: it applies generated patches and runs repository tests in Docker. Its result categories distinguish an evaluation producing a report from a patch resolving the issue. Those counts answer different questions.
OpenAI’s October 6, 2026 Ironclad research illustrates another distinction. Its eleven workflow tasks were evaluated against detailed criteria, with mean rubric scores and simulated time estimates. Those measurements describe research performance. They do not directly supply a count of fully accepted workflows or measured human repair time.
The work left unfinished
Consider two options attempting the same ten tasks under the same allowed budget. These figures are illustrative.
| Option | Total attempt cost (USD) | Accepted tasks out of 10 | Cost per accepted task |
|---|---|---|---|
| A | $10 | 5 | $2.00 |
| B | $12 | 8 | $1.50 |
Cost per accepted completion = total cost of all attempts ÷ accepted completions.
Include failed attempts and retries in the numerator. Count each accepted task once. With zero accepted tasks, report that outcome and the spending; the ratio is undefined.
Option B has the lower ratio even though its total spending is higher. That makes it a better cost choice only if the accepted results are interchangeable for your purpose. A system that solves eight easy tasks but misses both urgent ones may be less useful than one that solves the urgent pair. Report results by task type and compare the same task IDs; the aggregate ratio can hide which work remains unfinished.
Report cost and time
Report elapsed time, retry counts and human review time separately. Specify which model, tool or other charges the total covers. Usage multiplied by rates is an estimate unless reconciled with attributable billing records. Missing cost coverage should stay visible.
Time adds a separate constraint. If a cheap workflow misses a deadline, its low cost per accepted task does not rescue the result. Set the minimum completion rate and maximum time first, then compare costs among workflows that meet them. If none meets them, the useful finding is that the workflow needs changing.
Keep a task-level record:
Task ID | Required outcome | Accepted? | All attempt costs
Elapsed time | Human review time | Unresolved reasonLeave unavailable measurements blank. Choose a workflow that meets the required outcome and deadline, then compare its cost per accepted task. Keep the record so a low average cannot hide the work that remains.