Evaluating agents

Evaluating an agent means measuring its behavior on a fixed set of tasks under a fixed procedure, so that results can be compared across checkpoints, models or harnesses. Training reward says whether optimization is succeeding against the reward. Evaluation asks what the policy can now do, and for agents the conditions it holds fixed include a harness, a runtime and budgets, each of which can move a score as much as a better checkpoint.

The evaluation protocol

Two scores are comparable only if they were produced the same way. The protocol of an agent evaluation includes:

the tasks and the split they come from
the harness, its version and its tool set
the runtime image and its network policy
sampling settings, which are part of the behavior policy
budgets: token limits, turn limits and timeouts
the number of rollouts per task
the scorer, including the judge model if one is used
for conversational tasks, the user simulator's model and prompt

A higher token limit lets more tasks finish, and a different harness changes what the model sees, so a report states whether it used a standard harness, which compares checkpoints, or the product harness, which measures what users get. A simulated user is a second sampled policy, so its randomness adds variance of its own, and changing its model can move the score with no change to the agent. Success belongs next to tokens, turns, wall time and cost, because a policy can raise its success rate by spending more of each.

pass@k and its unbiased estimator

pass@k is the probability that at least one of $k$ independent attempts at a task succeeds, averaged over tasks. pass@1 measures reliability, and larger $k$ measures coverage, meaning whether the policy can solve the task at all.

The standard estimator, from the HumanEval paper, draws $n \ge k$ samples per task, counts the $c$ that succeed, and computes the probability that a random subset of $k$ contains at least one success:

\text{pass@}k = 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}

The fraction is the probability that all $k$ chosen samples are failures. The estimator is unbiased, while the plug-in estimate $1 - (1 - c/n)^k$ is biased low. With $n = 10$ and $c = 3$ :

pass@1 $= 1 - 7/10 = 0.30$
pass@5 $= 1 - \binom{7}{5}/\binom{10}{5} = 1 - 21/252 \approx 0.917$
the plug-in estimate for $k = 5$ is $1 - 0.7^5 \approx 0.832$

Binomial coefficients overflow for large $n$ , so implementations use a product form:

import numpy as np

def pass_at_k(n: int, c: int, k: int) -> float:
    if n - c < k:
        return 1.0
    return 1.0 - np.prod(1.0 - k / np.arange(n - c + 1, n + 1))

The complementary pass^k, introduced with τ-bench, is the probability that all $k$ attempts succeed, estimated as $\binom{c}{k}/\binom{n}{k}$ . With $c = 3$ of $n = 10$ , pass^2 $= 3/45 \approx 0.067$ . It measures the consistency that matters for an agent users run many times. pass@k also differs from best-of-k: pass@k assumes an oracle that recognizes the success, while best-of-k needs a selector, such as a reward model or the agent's own tests, to pick it.

The two ends of the $k$ axis can move differently under RL. Yue et al. found RLVR models beating their base models at small $k$ while the base models reached higher pass@k at large $k$ , a result that longer training and stricter success criteria complicate (RL scaling). Reporting several values of $k$ shows which end moved.

Held-out tasks and environments

A held-out split of the training distribution tests whether gains transfer to instances that never provided gradient, and a held-out environment, with different tools, state or verifier, tests a stronger form of transfer. Training reward that rises while held-out performance stays flat points to memorized artifacts or reward hacking.

A split is only as independent as its construction. Tasks from the same repository, templates filled with different values, or near-duplicate problems leak across a random split, so splitting by repository, source or template measures transfer more faithfully. The suite, its primary metrics and its sampling budget should be fixed before the run, so the headline comparison is not selected afterward.

Contamination

A benchmark whose tasks appeared in pretraining data measures recall. Wu et al. gave Qwen2.5-Math-7B the first 60% of each MATH-500 problem and found it completed 54.6% of them word for word, against 0% on the newer LiveMathBench. Prefix-completion tests, benchmarks released after the model's data cutoff, and private task sets are the available checks.

What agent benchmarks fix

Public agent benchmarks differ in which parts of the protocol they fix:

Benchmark	Measures	Fixes	Leaves open
SWE-bench Verified	500 Python repository issues	human-screened specs and tests	public repositories, so contamination
SWE-Bench Pro	1,865 multi-file tasks from 41 repositories	held-out and commercial repositories	harness and budget choice
Terminal-Bench 2.0	89 command-line tasks, each with its own environment	human-written solutions and tests	at 89 tasks, a standard error near 5 points at 50% success (binomial estimate)
τ²-bench	support tasks where agent and user both act on shared state	task generator and constrained user simulator	user-simulator variance
BrowseComp	1,266 hard-to-find facts on the web	short answers checked against a reference	the live web changes and answers can leak
METR time horizon	human task length at 50% agent success	a unit comparable across models	depends on the task suite and human baselines

Variance and confidence intervals

An eval score is an estimate, and its uncertainty is often larger than the difference being reported. Treating the tasks as a sample from a larger population, as Miller recommends, gives a standard error for the mean over $M$ tasks. For $M = 200$ tasks with one binary attempt each and a success rate of $0.40$ :

\text{SE} = \sqrt{\frac{0.40 \times 0.60}{200}} \approx 0.035

A 95% confidence interval is about $\pm 1.96 \times 0.035 \approx \pm 0.068$ , or $[0.33, 0.47]$ .

More rollouts per task help less than expected. With $r$ rollouts per task, the variance of the mean splits into a part from differences between tasks and a part from sampling within each task:

\operatorname{Var}(\bar s) = \frac{1}{M}\left(\operatorname{Var}(p_i) + \frac{\mathbb{E}[p_i(1 - p_i)]}{r}\right)

where $p_i$ is the policy's true success rate on task $i$ . Suppose $\operatorname{Var}(p_i) = 0.15$ and $\mathbb{E}[p_i(1-p_i)] = 0.09$ , which add up to the total $0.24$ above. Going from $r = 1$ to $r = 4$ shrinks the standard error from $0.035$ to $0.029$ , and $r = 16$ only reaches $0.028$ . Most of the uncertainty comes from which tasks are in the set, and only more tasks reduce it. Rollouts per task are still needed for pass@k and per-task difficulty.

Two checkpoints evaluated on the same tasks call for a paired comparison: compute the per-task difference $d_i$ and its standard error $\sqrt{\operatorname{Var}(d_i)/M}$ . Task difficulty affects both checkpoints alike and cancels. If a new checkpoint goes from 80 to 90 successes on 200 tasks by fixing ten tasks and breaking none, the improvement is 0.05 with a paired standard error of about 0.015, a 95% interval of roughly $[0.02, 0.08]$ , even though each score's own interval is wider than 0.05.

Reading beneath the mean

An aggregate hides which tasks changed. Per-environment results belong in the report next to any macro average, and difficulty slices show whether a policy improved only where the base model already succeeded occasionally. The fractions of tasks where every rollout succeeds or every rollout fails show whether the frontier of partially solved tasks is moving, which connects evaluation to difficulty filtering. Error and truncation rates belong in the report, because a crash or a slow sandbox is not a wrong answer.

A small, stable set of traces reviewed across checkpoints separates failures that look identical in the score: a reward shortcut, a harness that hid needed information, an environment timeout, a verifier that rejected valid work, or an improvement bought with far more compute.

In prime-rl

The reverse-text example measures its baseline with uv run eval @ examples/basic/reverse-text/eval.toml: 20 tasks with 3 rollouts each give Qwen3-0.6B an average reward of about 0.05, and about 0.8 after SFT and 20 RL steps. uv run eval reuses the orchestrator's eval pipeline, so tasks, scoring and traces match training. The same sources can run inside an RL job against the current policy:

[orchestrator.eval]
interval = 10

[[orchestrator.eval.source]]
name = "reverse-text"
group_size = 8
env.taskset.id = "reverse-text"
env.agent.harness.id = "null"
env.agent.runtime.type = "subprocess"

Eval sampling is configured separately from training sampling. When every reward is 0 or 1, prime-rl reports the unbiased pass@k and pass^k above at powers of two up to group_size, over non-errored rollouts. reverse-text's partial-credit reward is not binary, so it gets the mean (avg@k) only.