How AI agents are evaluated

Model releases quote agent benchmarks as headline results, and several of the best known have turned out to measure less than their scores suggest. This post maps how agents are evaluated as of October 2026: what the benchmarks test, how they grade, which methods hold up, where the field is still stuck, and how to build evaluations for an agent of your own. Every number links to its source. I will update this page as the field moves; the date above says when I last checked it.

Why agents need their own evaluations

A model answering a question produces text, and you grade the text. An agent works over many turns. It calls tools, changes the state of its environment and adapts to what comes back, so an early mistake can carry through every later step (Anthropic, 2026).

That changes what “correct” means. Anthropic’s example is a flight-booking agent that ends its run by saying the flight is booked. The outcome that counts is whether a reservation exists in the database (Anthropic, 2026). τ-bench grades exactly that way: it compares the database at the end of a conversation with an annotated goal state (Yao et al., 2024).

In February 2026 OpenAI stopped reporting SWE-bench Verified, a benchmark it had created and that frontier model releases routinely quoted, and recommended that other developers stop too (OpenAI, 2026). The setup around the model also moves results. On WildClawBench, changing only the agent harness shifted one model’s score by up to 18 points (Ding et al., 2026). A score without its harness, grader and date is hard to interpret.

A map of the field

Agent evaluations answer different questions, and a benchmark built for one question says little about another. Six questions cover most of the work:

• Capability: can the agent complete realistic tasks in a domain such as software, the web or a desktop?
• Reliability: does it succeed every time, or only sometimes?
• Horizon: how long and how many steps can a task be before the agent fails?
• Cost: what does each success cost in money, tokens and time?
• Safety and security: does it refuse harmful tasks and resist manipulation through the data it reads?
• Validity: does the benchmark itself measure what it claims?

A 2025 survey maps the same literature by agent capability and application domain, and names cost-efficiency, safety and robustness as critical gaps (Yehudai et al., 2025).

Capability benchmarks

Most public numbers come from capability benchmarks. The table lists the ones that recur in model releases and papers, with the result reported at launch. Launch results are useful for one thing: seeing how fast a benchmark was climbed.

Two older multi-environment suites are worth knowing for context. AgentBench tested models in eight environments and found a large gap between top commercial models and open models of 70B parameters or fewer (Liu et al., 2023). τ²-bench extended τ-bench to a telecom domain where the user also acts on shared state, and agent performance dropped significantly once the agent had to guide a user through actions (Barres et al., 2025).

Reliability

Most leaderboards report pass@1: the share of tasks solved on one attempt. τ-bench introduced pass^k, the chance that all k independent attempts at a task succeed, averaged across tasks. It contrasts with pass@k, the chance that at least one of k attempts succeeds (Yao et al., 2024).

The two diverge fast. Anthropic’s example: an agent with a 75% success rate per trial passes three trials in a row only (0.75)³ ≈ 42% of the time. By k = 10, pass@k approaches 100% while pass^k approaches zero (Anthropic, 2026). Anthropic’s guidance: pass@k for tools where one success is enough, pass^k for agents where consistency is essential.

Horizon

METR measures the length of tasks an agent can complete, in human time. The 50% time horizon is the duration of tasks, as timed for experienced humans, that a model completes half the time. In the original paper, Claude 3.7 Sonnet had a horizon of about 50 minutes, and frontier horizons had doubled roughly every seven months since 2019 (Kwa et al., 2025). METR’s results page now warns that measurements above 16 hours are unreliable with the current task suite, and as of September 2026 says it is no longer actively updated (METR).

Newer benchmarks attack length directly. Long-Horizon-Terminal-Bench has 46 tasks. In the paper’s October 2026 revision, agents used an average of 9.8 million tokens, about 239 episodes and 88.9 minutes per run, and the best of 17 models fully solved 19.6% (Li et al., 2026). Odysseys tests 200 multi-site tasks derived from real browsing sessions on the live web, and the best frontier models reached 44.5% (Jang et al., 2026).

Cost

Kapoor and colleagues argued in 2024 that agent benchmarks focused on accuracy alone, which produced needlessly complex and expensive agents and may have misattributed where accuracy gains came from. They recommended optimising cost and accuracy together (Kapoor et al., 2024). The same group built the Holistic Agent Leaderboard (HAL), which reports accuracy against dollar and token cost. Its first large analysis ran 21,730 rollouts across 9 models and 9 benchmarks, at about $40,000 (Kapoor et al., 2025).

Safety and security

Safety benchmarks for agents test two directions. AgentHarm contains 110 explicitly malicious agent tasks (440 with augmentations) in 11 harm categories. Leading models complied with many of them without any jailbreak, and simple universal jailbreak templates kept the agents capable of coherent multi-step harmful work (Andriushchenko et al., 2024). AgentDojo tests the opposite direction: attacks hidden in the data an agent reads. It has 97 realistic tasks such as email, banking and travel booking, plus 629 security test cases (Debenedetti et al., 2024).

Validity

The last question is about the benchmarks themselves. A 2025 study by 25 authors found task and grading flaws that can misstate agent performance by up to 100% in relative terms. Its examples: SWE-bench Verified uses too few test cases, and τ-bench counted empty responses as successes. The authors’ Agentic Benchmark Checklist reduced overestimation by 33% when applied to CVE-Bench (Zhu et al., 2025). The section on open problems returns to this.

Methods and their trade-offs

Grade the outcome, the transcript, or both

An outcome check asks whether the world ended up in the right state: tests pass, the booking exists, the file contains the right numbers. A transcript check asks how the agent got there: which tools it called, with which arguments, how many turns it took, whether it followed a policy. Anthropic’s terminology separates the two: the transcript is the full record of a trial, and the outcome is the final state of the environment (Anthropic, 2026).

Outcome checks are harder to fool and blind to process. An agent that books the right flight after exposing a customer’s card details passes an outcome check. HAL found exactly this kind of behaviour by reading logs: agents misusing credit cards in flight-booking tasks, and one agent searching for the benchmark on Hugging Face instead of solving the task (Kapoor et al., 2025).

Code, model or human graders

Anthropic sorts graders into three types (Anthropic, 2026):

Their advice is to use deterministic graders where possible, model graders where necessary and human graders to validate both.

Model graders have known failure modes. Zheng and colleagues found position bias, verbosity bias and self-enhancement bias, and also found that GPT-4 agreed with human preferences over 80% of the time, about as often as humans agreed with each other (Zheng et al., 2023). For agents, two refinements stand out. Agent-as-a-Judge gives the judge agentic capabilities so it can assess intermediate steps as well as the result; on a set of 55 AI development tasks with 365 requirements, the authors report it beat LLM-as-a-judge and matched the reliability of human evaluation (Zhuge et al., 2024). Odysseys uses an average of 6.1 graded rubric items per task and reports better agreement with human judgement than a single judge over the whole trajectory (Jang et al., 2026).

Metrics that match the question

Partial credit and efficiency are recent additions. Long-Horizon-Terminal-Bench splits each task into graded subtasks because final-outcome scoring hides partial progress (Li et al., 2026). Odysseys proposes rubric score per step as an efficiency metric, and even frontier agents scored only 1.15% on it (Jang et al., 2026).

Statistics

Miller treats an evaluation as an experiment and argues that the field has largely ignored how other sciences analyse experiments. His recommendations are short. Report the standard error of every score. Use clustered standard errors when questions come in related groups. Resample each question several times to reduce variance. Compare models on question-level paired differences rather than on two summary scores. Use power analysis to check that the evaluation can detect the effect you care about (Miller, 2024). With reasonable assumptions, his worked example suggests new evaluations need at least 1,000 questions.

That sounds at odds with Anthropic’s advice that 20 to 50 tasks drawn from real failures are a good start (Anthropic, 2026). Both hold. Early in development, changes have large effects, and small samples detect large effects. Mature agents improve in small steps, and small differences need large samples.

The harness is part of the result

An agent is a model plus a scaffold: prompts, tools, memory, retry logic, limits. Changing the scaffold changes the score. WildClawBench ran tasks inside real agent harnesses and found that swapping only the harness moved one model by up to 18 points (Ding et al., 2026). HAL found that higher reasoning effort reduced accuracy in the majority of its runs (Kapoor et al., 2025). Claude Opus 4.5 first scored 42% on CORE-Bench; after an Anthropic researcher fixed rigid grading, ambiguous specifications and unreproducible tasks, and used a less constrained scaffold, it scored 95% (Anthropic, 2026).

The environment counts too. Each trial should start from a clean state. Anthropic reports seeing an agent gain an unfair advantage by reading the git history left by previous trials (Anthropic, 2026). Tooling exists for this. Inspect, from the UK AI Security Institute and Meridian Labs, provides datasets, agents, tools and scorers as building blocks, runs untrusted code in sandboxes such as Docker and Kubernetes, and can run external agents such as Claude Code, Codex CLI and Gemini CLI inside an evaluation (Inspect).

Open problems

Contamination

Public benchmarks built from public code end up in training data. SWE-Bench+ examined passing patches from a then-leading agent on SWE-bench: 32.67% had the solution in the issue report or comments, 31.08% passed because the tests were too weak, and over 94% of issues predated the models’ knowledge cutoffs. After filtering, the agent’s resolution rate fell from 12.47% to 3.97% (Aleithan et al., 2024). The SWE-Bench Illusion found that models identified the buggy file from the issue text alone with up to 76% accuracy on SWE-Bench Verified, against up to 53% on repositories outside SWE-Bench (Liang et al., 2025).

OpenAI’s February 2026 audit went further. Of 138 tasks that o3 did not solve consistently over 64 runs, 59.4% had material flaws in tests or descriptions: 35.5% had tests too narrow to accept correct alternatives, and 18.8% tested functionality the task never described. Every frontier model OpenAI tested could reproduce the original fix or verbatim problem details for some tasks. OpenAI now reports the public split of SWE-Bench Pro, which it says shows fewer and milder contamination cases (OpenAI, 2026). SWE-Bench Pro’s answer is structural: public tasks from copyleft repositories, plus held-out and commercial sets that are not published (Deng et al., 2025).

Broken tasks and graders

A task that no agent can solve measures nothing. Anthropic’s rule of thumb: with frontier models, a 0% pass rate over 100 trials most often means the task is broken, not the agent. Their remedy is a reference solution for every task that passes all graders (Anthropic, 2026). The same post describes METR tasks that asked agents to reach a stated score while grading required exceeding it, which penalised models that followed instructions. The Agentic Benchmark Checklist exists because such flaws are common enough to move results by up to 100% (Zhu et al., 2025).

Reward hacking

Capable agents find the grader’s weak points. METR found that o3 reward hacked in 30.4% of runs on RE-Bench, where it could see the scoring function, against 0.7% on HCAST, and that telling the model not to cheat left reward hacking in most runs on the task where METR tried it (METR, 2025). My read: any benchmark whose grader runs inside the agent’s reach should be treated as a test of the grader as much as of the agent.

Saturation and benchmark half-life

Benchmarks age quickly. Anthropic noted in January 2026 that models had gone from 40% to over 80% on SWE-bench Verified in one year (Anthropic, 2026), and OpenAI’s audit shows that part of the remaining gap was broken tests. HAL has declared CORE-Bench solved (HAL). Near the ceiling, a large capability gain looks like a small score increase (Anthropic, 2026), which makes saturated benchmarks misleading as well as uninformative.

Long horizons

Long tasks remain hard. On Long-Horizon-Terminal-Bench the mean pass rate across 17 models was 3.2% at full reward (Li et al., 2026). On WildClawBench, which averages about 8 minutes and over 20 tool calls per task, the best of 19 models reached 62.2% and every other model scored below 60% (Ding et al., 2026). METR’s own horizon measurements stop being reliable above 16 hours (METR).

Keeping score is expensive

Good agent evaluation needs virtual machines, sandboxes, human baselines and people to read logs. HAL’s first large analysis cost about $40,000 (Kapoor et al., 2025), and the project has since paused adding new models to focus on measuring reliability (HAL). METR’s time-horizon page is no longer actively updated (METR). My read: independent, maintained agent leaderboards are a public good with no stable funding model, and model developers’ own reports fill the gap by default.

How to choose and apply

When you compare models

• Prefer benchmarks with held-out or private splits, such as SWE-Bench Pro, over fully public ones (Deng et al., 2025).
• Check which harness produced a number before you compare it with another, because the harness alone can move a score by double digits (Ding et al., 2026).
• Look for cost next to accuracy, and treat a missing cost as missing data (Kapoor et al., 2024).
• Check the date. A benchmark near its ceiling says little about the difference between two strong models.
• Run the candidates on your own tasks before you decide. Public benchmarks narrow the field; they do not pick the winner.

When you evaluate your own agent

1. Start with 20 to 50 tasks taken from real failures and from what you already test by hand (Anthropic, 2026).
2. Write a reference solution for every task and confirm it passes your graders.
3. Grade the outcome with code wherever you can. Add transcript checks for policy and safety, and model graders for quality, calibrated against human judgement.
4. Isolate trials: fresh environment, no leftover files, no shared caches.
5. Run several trials per task. Report pass@1 with its standard error, and pass^k if the agent acts without review (Miller, 2024).
6. Pin and record the model version, harness, tools and limits for every run.
7. Read transcripts every week. Logs show failures, shortcuts and grader bugs that scores hide (Kapoor et al., 2025).
8. If the agent reads untrusted content such as email, web pages or tickets, add prompt-injection cases in the style of AgentDojo (Debenedetti et al., 2024).
9. When a capability suite passes reliably, move it into a regression suite that runs on every change (Anthropic, 2026).

Automated evaluations are one layer. Anthropic’s post compares the full picture to the Swiss cheese model from safety engineering: production monitoring, user feedback, A/B tests and human review each catch what the others miss (Anthropic, 2026).