---
title: "AI agent benchmarks: what they test and what they miss"
description: "AI agent benchmarks compared: GAIA, WebArena, OSWorld and more. What their tasks and leaderboards measure, and where personal-agent evidence is missing."
url: https://personalagentbench.com/benchmarks/
last_updated: 2026-10-16
---

# AI agent benchmarks: what they test and what they miss

AI agent benchmarks measure different things, from retrieving a fact to changing a browser or desktop state. A personal agent benchmark is hard to find, which is the short answer to the question most people arrive here with. Agent benchmarks are not in short supply, and the nine on this page are serious work, but almost none of them measure a personal agent. The reason is structural rather than an oversight somebody could patch in the next release: they hand an agent a task in a world built for the test, at most with some seeded history, and a personal agent is defined by a continuing relationship with one person and access to things of theirs it could break.

This site is an attempt at the other kind. The scorecard on the [front page](https://personalagentbench.com/) runs ordinary errands through adapted product runtimes in resettable simulated worlds, with the exam's own model substituted for each product's. It publishes a rate, an interval and a task count wherever a result is usable, and individual attempts are published as [records](https://personalagentbench.com/runs/) for Instinct, Muse, Pine and Town. This page is about the nine benchmarks it is not, because knowing what they already answer is the only way to say what is left over.

- Some benchmarks, including OSWorld 2.0, supply seeded user-history context. Such task context does not by itself measure learning a continuing user's preferences across errands.
- Three focus on information retrieval; the others include actions under benchmark-specific environments, with different coverage of consequential side effects.
- Their numbers vary wildly in how checkable they are: GAIA publishes every submission through an API, BrowseComp has no open submission board, and WebArena and AgentBench keep their scores in spreadsheets.
- None of this makes them bad benchmarks. It makes them benchmarks for a different question than the one you're asking when you pick an assistant.
- There are research benchmarks for personalisation and memory, and the consumer evaluations described below address parts of that space; this site adds runtime observations and a controlled simulated exam.

## What each agent benchmark measures, and what it misses

| Benchmark | What it measures | What it can't tell you |
| --- | --- | --- |
| [GAIA](https://personalagentbench.com/benchmarks/gaia/) | Assistant-style questions that need several steps, a browser and sometimes a file to answer. | Every question arrives cold. Nothing carries over from the last one, so a system that remembers you scores exactly the same as one that doesn't. |
| [BrowseComp](https://personalagentbench.com/benchmarks/browsecomp/) | Finding facts that are genuinely hard to find, which mostly tests whether an agent keeps digging. | Reading, not doing. Nothing is booked, cancelled, paid for or sent. |
| [WebArena](https://personalagentbench.com/benchmarks/webarena/) | Completing tasks inside a browser on self-hosted sites, so the environment stays identical between runs. | The sites aren't yours and hold nothing about you. No saved cards, no order history, no account that remembers. |
| [OSWorld](https://personalagentbench.com/benchmarks/osworld/) | Driving a real desktop through long professional workflows, graded on the state of the machine afterwards. | Skill, not memory. It asks whether an agent can operate the machine, never whether it knows why you'd want it to. |
| [Tau-bench](https://personalagentbench.com/benchmarks/tau-bench/) | Holding to a domain's rules across a conversation with a user, and doing it consistently when the same task is run again. | The user is simulated and the domains are narrow, but this is the closest thing on the list to what a personal agent actually does. |
| [AgentBench](https://personalagentbench.com/benchmarks/agentbench/) | One agent across several unrelated environments, which is a decent proxy for general competence. | None of the environments is your life. No calendar, no inbox, no bill to dispute. |
| [Terminal-Bench](https://personalagentbench.com/benchmarks/terminal-bench/) | Work at a command line, where success is unambiguous and cheap to check. | Almost nothing a personal agent does happens at a terminal. |
| [SWE-bench](https://personalagentbench.com/benchmarks/swe-bench/) | Fixing real GitHub issues, graded by the repository's own test suite rather than by a judge. | A closed problem with a written specification. An errand has neither, and it cannot be rolled back. |
| [AssistantBench](https://personalagentbench.com/benchmarks/assistantbench/) | Realistic web tasks that take a while, which is closer to an errand than most of this list. | Still one-shot. No standing preferences, no routine that fires next Monday. |

Qualitative only, on purpose. Leaderboard numbers and task counts belong on the per-benchmark pages, and those pages ship once the numbers are verified against the source rather than a summary of it.

Pages written so far: [GAIA](https://personalagentbench.com/benchmarks/gaia/), [BrowseComp](https://personalagentbench.com/benchmarks/browsecomp/), [WebArena](https://personalagentbench.com/benchmarks/webarena/), [OSWorld](https://personalagentbench.com/benchmarks/osworld/), [Tau-bench](https://personalagentbench.com/benchmarks/tau-bench/), [AgentBench](https://personalagentbench.com/benchmarks/agentbench/), [Terminal-Bench](https://personalagentbench.com/benchmarks/terminal-bench/), [SWE-bench](https://personalagentbench.com/benchmarks/swe-bench/), [AssistantBench](https://personalagentbench.com/benchmarks/assistantbench/). Every leaderboard number on them is pulled from the source rather than copied off a screenshot.

The middle column is what each benchmark's authors set out to measure, and most succeeded. The right-hand column is what the design can't see, which is not a criticism of it.

## No agent benchmark here follows one user over time

Read the right-hand column again and the same sentence keeps turning up in different words.

GAIA, WebArena and BrowseComp start each task without a continuing relationship with the user. OSWorld 2.0 goes further and seeds some tasks with user history, but that history is written for the task rather than built up by the agent. They can measure task performance under those setups, but not the value of remembering a particular person.

Consider what that erases. A standing preference is worth zero points. So is a correction you made last week that should have stuck, or a routine that fires every Monday without being asked.

For a research benchmark that's the right call. Holding memory constant is the simplest way to compare two systems fairly, because otherwise you're comparing setups rather than agents. For choosing a product you'd hand your accounts to, it removes the one property that separates a personal agent from a very good search box.

## Finding things and doing things are different tests

The second split runs through this list and it's the one people conflate most often.

GAIA and BrowseComp are retrieval. Every answer is a short fact checked against a reference: GAIA by quasi exact match after normalisation, BrowseComp by a grader model that decides whether the final answer matches. WebArena, OSWorld and the ALFWorld-style environments inside AgentBench are execution: the agent changes something and the change is inspected afterwards. Those two fail in ways that are not comparable.

A wrong retrieval hands you a wrong answer and you notice. A wrong action charges a card, cancels something you meant to keep, sends a message to a person who shouldn't have received it, or fills a form with the wrong address. The cost of being wrong isn't symmetrical, and a benchmark made entirely of retrieval never has to price it.

This is why a headline number quoted without its benchmark is close to meaningless. "Scores 93% on agent tasks" may mean it answers research questions well, and says nothing about whether it should be allowed near your bank.

## An agent benchmark that passes its human baseline has finished its job

A benchmark is a ruler, and it works for as long as the things being measured still fit on it.

GAIA is the clearest case. At launch, the best system managed 15% against a human baseline of 92%, and every point of improvement meant something. As of 2 November 2026 the top self-submitted entry scores 94.02%, above the human number the paper set as its target, with the leaders clustered within about a point of each other. The gap the benchmark existed to measure is gone, at least on the submitters' own setups; Princeton's HAL, which reruns agents itself on the public validation questions, tops out at 74.55%.

WebArena has now crossed its human figure too, at least on paper. Its best GPT-4 agent scored 14.41% in the original paper against a human 78.24%. When we read the [leaderboard sheet](https://docs.google.com/spreadsheets/d/1M801lEpBbKSNwP-vDBkC_pF7LdyGU1f_ufZb_NWNBZQ/edit) on 16 October 2026, its top row was [Narada AI](https://personalagentbench.com/narada-ai/) Code at 86.1%, dated September 2026 and marked as not having released its code or trajectories. The highest entry that did release them was at 74.3%. So the claim that agents have passed humans on WebArena rests, for now, on one submission that has not released the material anyone else would need to check it.

Human baselines deserve a closer look before you lean on them. WebArena's 78.24% comes from five computer science graduate students doing one sampled task from each of 170 templates, not the 812 tasks the agents are scored on. The leaderboard sheet flags this in a note that is rarely quoted alongside the number. A gap to a human baseline is a useful signal, not a measurement of the same thing on both sides.

On GAIA the ruler has run out, and treating a saturated leaderboard as a live ranking is how a two-point difference gets mistaken for a finding.

## Where agent benchmark numbers actually come from

The most useful question to ask about any leaderboard is who computed the number and whether you can reproduce it. The answers on this page differ more than you'd expect. These are the official boards our pages read; each benchmark page dates its own snapshot.

- GAIA: [results dataset on Hugging Face](https://huggingface.co/datasets/gaia-benchmark/results_public), 16 October 2026.
- BrowseComp: [Kaggle's board](https://www.kaggle.com/benchmarks/openai/browsecomp), 16 October 2026.
- WebArena: [leaderboard sheet](https://docs.google.com/spreadsheets/d/1M801lEpBbKSNwP-vDBkC_pF7LdyGU1f_ufZb_NWNBZQ/edit), read on 16 October 2026 for the top row quoted above.
- OSWorld 2.0: [osworld-v2.xlang.ai](https://osworld-v2.xlang.ai/).
- tau-bench: [taubench.com](https://taubench.com).
- Terminal-Bench: [tbench.ai](https://www.tbench.ai/).
- SWE-bench: [swebench.com](https://www.swebench.com/).
- AgentBench: [FC sheet](https://docs.google.com/spreadsheets/d/e/2PACX-1vRR3Wl7wsCgHpwUw1_eUXW_fptAPLL3FkhnW_rua0O1Ji_GIVrpTjY5LaKAhwO-WeARjnY_KNw0SYNJ/pubhtml), 16 October 2026.
- AssistantBench: [Hugging Face Space](https://huggingface.co/spaces/AssistantBench/leaderboard), figures from 8 October 2026; down with a runtime error on 16 October.

One independent alternative cuts across several of them. Princeton's [Holistic Agent Leaderboard](https://hal.cs.princeton.edu/) reruns agents itself on GAIA's validation set, AssistantBench's development tasks, TAU-bench airline, a SWE-bench Verified mini set and others, and prints the cost of each run. It said on 16 October 2026 that it has paused adding new models.

GAIA publishes every submission as [a dataset](https://huggingface.co/datasets/gaia-benchmark/results_public) you can pull through an API, so our page shows a table we paged and sorted ourselves, dated to the day. BrowseComp has no open submission board we could find: there is OpenAI's launch evaluation, [a Kaggle-run board](https://www.kaggle.com/benchmarks/openai/browsecomp) last updated in March 2026 and unchanged when we re-read it on 16 October, and vendor self-reports that now run above 90%. WebArena's official leaderboard is [a Google spreadsheet](https://docs.google.com/spreadsheets/d/1M801lEpBbKSNwP-vDBkC_pF7LdyGU1f_ufZb_NWNBZQ/edit) linked from the project page, with a column recording whether each submission released its code and trajectories. AgentBench keeps its current results in [a published sheet](https://docs.google.com/spreadsheets/d/e/2PACX-1vRR3Wl7wsCgHpwUw1_eUXW_fptAPLL3FkhnW_rua0O1Ji_GIVrpTjY5LaKAhwO-WeARjnY_KNw0SYNJ/pubhtml) too, and its original leaderboard is a screenshot in a repository README.

There's a second thing to look for, which is who the top entries belong to. On AgentBench's current leaderboard the first five rows are all AgentRL models, listed under Z.ai and trained with the reinforcement learning framework from the same GitHub organisation that maintains the benchmark, and the highest-placed frontier API model sits sixth. Nothing about that is improper, and 23 of the sheet's 25 rows cite the maintainers' own AgentRL work as their result source, which is better for comparability than accepting self-reports. The organisational relationship is documented. It does not establish how training exposure caused the ordering.

A score with no shared scoring harness behind it isn't a measurement, it's a claim, and the distance between the two is where most agent marketing lives.

## Two consumer assistant scorecards, and why not to average them

Everything above is a research benchmark. Two public efforts do test consumer personal agents directly, and both are worth reading. They are also a clean demonstration of why two scores for the same product can't be combined.

[Assistant Benchmark](https://assistantbenchmark.com/), which is not the academic AssistantBench mentioned below, is run by David Pawlan, who its About page says tests the assistants from his own accounts and sets every score. Each category is scored out of 10, every score links to its own evidence, and an untested cell is left blank rather than guessed. When we read it on 22 October 2026, its general board, last updated 19 October, gave an overall score to 29 assistants across up to 15 scored categories and ranked them. Muse was first at 9.2 with 8 categories tested, and Instinct fourth at 8.3 with 12 tested, so the two rankings rest on different sets of blanks.

[RuntimeWire](https://runtimewire.com/article/ai-assistant-showdown-grok-bot-instinct-claude-chatgpt-work-muse), a publication that covers the AI industry, ran what it calls its first assistant bake-off on 16 September: one evening, a nine-task pack, five products, one AI model as judge, and a working index out of 5 after three unscored dimensions were dropped and the rest reweighted. It tested Instinct over iMessage and Muse on the web. Instinct came out at 3.55 and Muse at 2.46.

So one board puts Muse above Instinct and the other puts Instinct above Muse. Neither is obviously wrong. Averaging them would be:

- One is marked by one person out of 10 from real use on real accounts, the other by a model out of 5 in one evening.
- The tasks differ, and even within one board the two products were scored on different subsets of them.
- The surfaces differ. A product reached through iMessage and a product reached on the web are not the same test.

A blended number would hide all three and print a precision neither source claims. A score is how one product did on one set of tasks for one tester, not how many people use it or keep using it.

## Personal agent benchmarks that do exist, and what they still skip

The nine in the table are the general agent benchmarks people quote. Search for a personal agent benchmark specifically and a different shelf turns up: research papers that test personalisation, memory and proactivity head on. They are real work on exactly the gap described above, and worth knowing by name. One adjacent project is [PERSONA](https://arxiv.org/abs/2407.17387), whose PERSONA Bench measures alignment with 1,586 synthetic personas' preferences; it is about pluralistic alignment rather than completing errands. The rest are closer.

- [PersonaLens](https://arxiv.org/abs/2506.09902) (June 2025) gives assistants user profiles with preferences and interaction histories, then has one model play the user and another judge personalisation, response quality and task success.
- [MemoryAgentBench](https://arxiv.org/abs/2507.05257) (July 2025) tests memory itself: accurate retrieval, learning during the test, long-range understanding and selective forgetting, fed in over many turns rather than all at once.
- [Persona2Web](https://arxiv.org/abs/2602.17003) (February 2026) describes itself as the first benchmark of personalised web agents on the open web. Queries are deliberately ambiguous and the agent is meant to resolve them from the user's history.
- [π-Bench](https://arxiv.org/abs/2605.14678) (May 2026) is 100 multi-turn tasks across five user personas, with hidden intents and continuity between sessions, and it scores whether an assistant meets a need before it is stated.
- [Proactivity-Gym](https://arxiv.org/abs/2609.37267) (September 2026, KAIST and the University of Minnesota) grades an agent for acting on its own: 7 to 10 simulated days of calendar, message and file events, scored on finishing the work, doing what's due first, and choosing whether to suggest or just act. Its results cover models inside harnesses, OpenClaw among them, not shipping products, and its data isn't public yet. We use its framework on [proactive AI agents](https://personalagentbench.com/proactive-ai-agents/).
- [TraceDance](https://arxiv.org/abs/2609.33295) (September 2026, ByteDance and the University of Illinois Chicago) builds tests from 252,557 real Claude Code and OpenClaw sessions, each cut just before the agent went wrong, such as leaking a credential or pressing on after the user said no, and grades the model's next move. Nine models passed 26.7% on average and 6.9% on protecting secrets. It scores one step from a model, not a product; most sessions came from agents running ByteDance's Doubao models, and neither the sessions nor the code is public.
- [SovereignPA-Bench](https://arxiv.org/abs/2607.05363) (July 2026) is the nearest to our own errands: 1,920 booking and 288 cancellation scenarios on a scripted platform that nudges, upsells and over-collects, scoring whether the agent follows the user and asks before exceeding its authority.
- [Gaia2](https://huggingface.co/blog/gaia2) (September 2025), GAIA's successor from Meta and Hugging Face, replaces read-only questions with a mock smartphone holding email, calendar, contacts and shopping apps, so the agent has to change things as well as find them.
- [micro1's PersonalAgentBench](https://www.micro1.ai/personal-agent-bench) (no relation to this site despite the name) does run shipping products: six assistants, including Dots, Gemini Spark, Instinct and Muse, on 20 tasks in real accounts, one scored run each, judged by people on task completion and on staying within the user's authority, then ranked.

The papers each close part of the gap, but their subjects are models or research agents. Some use simulated worlds or users, while Persona2Web operates on the open web; these settings do not all share the same limits. micro1's test is closer to ours in subject, but at 20 tasks and one run each its widest gap is 80% against 55%; overlapping marginal intervals alone do not test significance. Paired outcomes and repeat measurements are needed for a justified comparison.

Of the nine in the table, tau-bench comes closest: it asks whether an agent holds to a domain's rules across a conversation and does the same thing when the task is run again, which is nearer to managing a calendar than raw capability is. AssistantBench's long, realistic web tasks are nearer to an errand than a quiz question. Neither carries anything from one task to the next.

## Why we built a personal agent benchmark of our own

Every row in the table above shares a blind spot, and for a research benchmark that's correct. For judging a product you'd give your email, your calendar and a payment method to, it's disqualifying.

So our scorecard asks different questions, and the [errands](https://personalagentbench.com/tasks/) behind them are ordinary ones rather than research puzzles, such as getting a subscription cancelled. Where does the model run and what can it reach on your machine. How does a tool definition get in front of it and who decided which ones. Where is your memory kept and can you correct it. What do you get to approve, and how do you reach it. And what could we not establish. Those are answerable today by observation, without waiting for a leaderboard, and they're the properties you can't change after you've committed.

Individual attempts are published as [recorded runs](https://personalagentbench.com/runs/) for four products, 245 of them when we counted on 16 October 2026, rather than only as an aggregate. The scorecard carries completion rates from the task runs, each with its interval and the count of tasks behind it, and a row whose figure doesn't reconcile with the records we can check carries no number at all rather than a number with a hedge next to it. What it still doesn't carry is one score per product. Everything above about provenance and sample size applies to our own table too, which is the uncomfortable half of writing this page.

## FAQ

**What are AI agent benchmarks?** Standardised task sets that measure whether a model can do something rather than say something: answer a multi-step research question, complete a purchase on a website, drive a desktop application, hold to a policy through a conversation. Each one defines its own environment, its own tasks and its own scoring.

**Which agent benchmark is best for choosing a personal assistant?** None of the nine in the table, which is an unsatisfying answer and an accurate one. None of them follows one user across tasks, so the memory and personalisation that make a personal agent worth having earn no points anywhere; even OSWorld 2.0's seeded history is written for the task.

**Why do the same models get very different scores on different benchmarks?** Because the benchmarks measure different failures and are run on different harnesses. GAIA has a human figure of 92% and a self-submitted leaderboard above it. On BrowseComp, human trainers solved 29.2%, Kaggle's best model run reached 31.2%, and vendors report above 90% on their own setups. The harness, budget and grader can move a score as much as the model does.

**Are agent benchmark leaderboards independently verified?** Rarely, and it varies per benchmark. Some publish every submission with its trajectories, some accept self-reports with no shared harness, and some have no open leaderboard. Each of our benchmark pages states which situation applies and links the source it was read from.

**Does any benchmark test whether an agent remembers me?** The nine do not establish a continuing relationship with a real user, though some include seeded history. MemoryAgentBench, PersonaLens and Persona2Web test aspects of memory and personalisation on research systems; Persona2Web uses the open web, so these are not all simulated-world evaluations.

**Can agent benchmark scores be gamed?** Yes, and it has been shown. In April 2026 a Berkeley team [described an exploit agent](https://rdi.berkeley.edu/blog/trustworthy-benchmarks-cont/) that reached near-perfect scores on eight well-known agent benchmarks without solving the tasks, for example every Terminal-Bench task through a fake `curl`, and about 98% on GAIA's validation questions, whose answers are public; its own results table lists 73% for OSWorld, so not every exploit was near-perfect. A score is only as trustworthy as the harness that computed it.

**Why doesn't this site just publish benchmark scores for the four agents?** A model's GAIA score does not establish a consumer product's performance. Our own published rates also have a limited scope: adapted product runtimes using a substituted exam model in simulated worlds, not consumer products tested end to end.

## What agent benchmarks leave for somebody else to answer

These leaderboards measure task performance. They do not settle whether a consumer product should hold access to your accounts.

For detail, see the [scorecard](https://personalagentbench.com/) that asks the other question, [how we score](https://personalagentbench.com/how-scoring-works/) for where its axes came from, and the four [teardowns](https://personalagentbench.com/agents/) they were built out of.

---

Published by Personal Agent Bench (https://personalagentbench.com), run by the Zentor team. Zentor builds agent software of its own, so its products are not on the scorecard and will not be added.