AgentBench: eight environments, a relative score, and no human baseline

AgentBench was the first benchmark to ask one question across many unrelated worlds: can this model act as an agent at all, or is it only good at the one environment somebody tuned it for. The answer arrives as a single number, and that number is stranger than it looks.

• Eight environments in the original paper, covering an operating system, a database, a knowledge graph, two games, a household simulator, a shopping site and web browsing.
• 1,014 problems in the test split, each needing several rounds of interaction, at roughly 11,000 model calls for a full run.
• The headline score is normalised against the 29 models tested in 2023, so a 4.01 means four times that cohort's average rather than 4.01 of anything.
• There's no human baseline anywhere in it, which makes the leaderboard a ranking of models against each other and nothing else.

What AgentBench actually is

Eight interactive environments, wired to one evaluation harness, with a model dropped into each of them in turn.

Three are grounded in code. The operating system task puts the model at a bash prompt in an Ubuntu container and asks it questions with deterministic answers, or gives it jobs like recursively making a directory tree read-only. The database task hands it a real MySQL instance. The knowledge graph task points it at Freebase, which has tens of millions of entities, and grades the answer by F1 because partial credit is the only sensible thing there.

Two are games: a digital card game, and lateral thinking puzzles where the model asks yes/no questions to reconstruct a story. Three were recompiled from existing work: ALFWorld for household tasks, WebShop for shopping, and Mind2Web for web browsing.

The test split is 1,014 problems and the development split is 269. That's deliberately modest, because every problem takes several rounds. The paper estimates five rounds for a database query and thirty-five for a household task, which works out at roughly 11,000 model calls for one full evaluation. AgentBench costs real money to run, and that constraint shaped its size.

The overall score is relative to the models of 2023

Here's the part that trips people up. AgentBench's headline figure, which the paper calls OA, is not a percentage and not a success rate. GPT-4 scored 4.01. Nothing is 4.01 out of anything.

The authors had a genuine problem to solve. The eight environments use five different metrics and produce wildly different score distributions: WebShop hands out reward generously, lateral thinking puzzles almost never do. Averaging the raw numbers would have made the benchmark mostly a WebShop benchmark with decorations. So each environment's scores were rescaled so that the average across all the models they tested came to 1, and the overall score is the average of those rescaled values.

That's a reasonable fix with a consequence nobody mentions when quoting the number. The scale is anchored to the 29 models evaluated in 2023, and those weights were then frozen so later work could reuse them. A model scoring 4.01 is scoring four times the 2023 cohort's average. It's a rank within a historical field, dressed as an absolute measurement, and it cannot be compared across benchmarks or read as "got 4% of something".

For the record, the frozen weights themselves say something. Lateral thinking puzzles got a weight reciprocal of 3.5 and WebShop 30.7, meaning the puzzles were about nine times harder for that cohort than the shopping tasks were.

Where the AgentBench leaderboard stands

The current leaderboard belongs to AgentBench FC, a function-calling rewrite announced in October 2025. It reports plain pass@1 per environment plus an unweighted average, so unlike the paper's OA these numbers are percentages and mean what they appear to mean.

Look at the column headed "Reported by" before the scores. Most rows were run by the maintainers themselves on a common harness, which is better for comparability than a leaderboard of self-reports, and the Claude row in sixth carries no stated source at all.

Then look at the organisation column. The top five entries are all AgentRL models, and AgentRL is the reinforcement learning framework from the same GitHub organisation that maintains AgentBench; the current version of the benchmark is built on it. The highest-placed model from anywhere else is Claude Sonnet 4.5 in sixth, at 58.9. None of that is improper and none of it is hidden. It does mean the ordering reflects who trained hardest against these five environments, which is a different claim from who is most capable, and a reader skimming for the top row will take away the second.

The benchmark shrank from eight environments to five

AgentBench FC covers ALFWorld, the database, the knowledge graph, the operating system and WebShop. The card game, the lateral thinking puzzles and Mind2Web are not in it.

That is a defensible engineering decision. Containerising an environment so anyone can reproduce a run is real work, and the three that were dropped are the three least like a practical agent task. But it means the sentence "AgentBench evaluates agents across 8 distinct environments", which is in the paper's abstract and in most descriptions of the benchmark, no longer describes what the live leaderboard measures. Two of the dropped environments were the game-grounded ones, so the benchmark's original claim to breadth is the part that got trimmed.

It also means the paper's numbers and the leaderboard's numbers are two different scales measuring two different task sets. You cannot put GPT-4's 4.01 next to Claude Sonnet 4.5's 58.9 and conclude anything.

Nobody has established what a person scores

The paper reports no human baseline and the leaderboard shows none.

For some of these environments that's understandable, since asking a person to play a digital card game against a scripted opponent is not obviously informative. For the operating system and database tasks it's a real absence, because those are things people do at work and a competent engineer's score would immediately tell you whether 51.7% is close to the ceiling or nowhere near it.

Without it, every AgentBench figure is relative. The paper's OA is relative to a 2023 model cohort. The FC average is relative to whichever models have been submitted. You can say one model beat another, and you cannot say whether the winner is any good.

Compare GAIA, which set a human baseline of 92% and was therefore able to announce when it had been passed, or WebArena, whose 78.24% gave three years of submissions something to aim at. AgentBench has a leaderboard with a top. It has no finish line.

What AgentBench can't tell you about a personal agent

None of the eight environments is your life.

There's no calendar in it, no inbox, no bill to dispute, no contact who needs telling you'll be late. The environments were chosen to be diverse and gradable, and diversity in that sense means unrelated to each other, not representative of anything you'd delegate.

And as with every benchmark on this bench, each problem starts cold. The model arrives at the database knowing nothing about which tables anyone cares about, and leaves without keeping anything. Memory, standing preferences and anything scheduled are worth zero points across all eight environments, which is the gap every leaderboard here shares.

The framing differs too, in a way worth noticing. AgentBench evaluates models, not products. What ships to you is a model plus a scaffold, a tool set, a memory system and a set of decisions about what it's allowed to touch without asking, and all of that is outside the frame. A model that scores well here tells you the raw material is capable. It tells you nothing about what was built on top of it.

Where these numbers came from

The leaderboard table is the published AgentBench FC sheet linked from the project README, exported as CSV and sorted by its own average column on 22 September 2026. The sheet titles itself as updated 18 November 2025 and no row in it is newer, so treat the ranking as roughly ten months old rather than current.

The paper figures, including the 1,014 test problems, the eight environments, the 29 models and how OA is computed, come from the paper itself, and we checked its arXiv page by its own title so we know we read the paper rather than a description of it. The list of environments in the current version comes from the repository README.

One detail that says something about the benchmark's stage of life. The original leaderboard, the one covering all eight environments, is published as an image file in the repository. There is no machine-readable version of it, so any table of those numbers you find elsewhere was transcribed by hand from a screenshot, including the few figures we quote above.

FAQ

What is AgentBench? A benchmark that evaluates large language models as agents across several interactive environments rather than on a single task type. The original version used eight environments and 1,014 test problems; the current function-calling version uses five.

What does an AgentBench score of 4.01 mean? It means roughly four times the average of the 29 models the authors tested in 2023. The paper's overall score is normalised against that cohort rather than being a percentage, so it can't be compared with scores from other benchmarks or read as a success rate.

Is there an official AgentBench leaderboard? Yes, as a published Google spreadsheet linked from the project README, covering the function-calling version. Its own title dates it to November 2025. The original eight-environment leaderboard exists only as an image in the repository.

Does AgentBench have a human baseline? No. The paper reports none and the leaderboard shows none, so every figure is a comparison between models rather than a measure of whether any of them is good enough to be useful.

How does AgentBench compare to WebArena or GAIA? AgentBench spreads one model across unrelated environments to test general competence. WebArena tests operating a website until the site's state is correct. GAIA tests multi-step research with an exact answer. AgentBench is the broadest and the least grounded in anything you'd personally want done.

Can a good AgentBench score tell me which assistant to use? No, and not only because of the cold start. AgentBench scores models, while a personal assistant is a model plus its scaffolding, tools, memory and permissions, and those are where products differ most.

What a personal agent scorecard has to ask instead

AgentBench answers "how broadly capable is this model when dropped into an unfamiliar interactive environment". That's worth knowing, and it's a property of the raw material rather than of anything you can buy.

The questions about a product are different. Where does the model run and what can it reach on your machine. How does a tool definition get in front of it, and who decided which ones. Where is your memory kept, and can you correct it when it's wrong. What can start the thing working while you're asleep. Those are the axes on our scorecard, which publishes completion rates with their intervals and the count of tasks behind them, and no combined score at all.

Read how we score, start with a teardown, or see what the other agent benchmarks measure.