---
title: "AgentBench: environments, relative scores and limits"
description: "AgentBench, the ICLR 2024 agent benchmark: eight environments, a relative score, its current five-environment leaderboard, and what it misses."
url: https://personalagentbench.com/benchmarks/agentbench/
last_updated: 2026-11-05
---

# AgentBench: environments, relative scores and limits

AgentBench is an agent benchmark from researchers at Tsinghua University, Ohio State and UC Berkeley, [first posted in August 2023](https://arxiv.org/abs/2308.03688) and published at ICLR 2024. Its authors present it as the first systematic benchmark to ask one question across many unrelated worlds: can this model act as an agent at all, or is it only good at the one environment somebody tuned it for. The paper's answer arrives as a single number, and that number is stranger than it looks. The current leaderboard uses a different, plainer one.

- Eight environments in the original paper, covering an operating system, a database, a knowledge graph, two games, a household simulator, a shopping site and web browsing.
- 1,014 problems in the test split, each needing several rounds of interaction, at roughly 11,000 model calls for a full run.
- The paper's headline score is normalised against the 29 models it tested, so GPT-4's 4.01 means about four times that cohort's average rather than 4.01 of anything.
- The live leaderboard is AgentBench FC: five environments, plain pass@1 rates, led by AgentRL with Qwen2.5-32B at 70.4% when we re-read it on 5 November 2026. There is no human baseline for either version.
- Not to be confused with agentbench.app, a 40-task benchmark for Claude Code setups, or eth-sri/agentbench, a coding-agent harness. Same name, different projects.

## What AgentBench actually is

Eight interactive environments, wired to one evaluation harness, with a model dropped into each of them in turn.

Three are grounded in code. The operating system task puts the model at a bash prompt in an Ubuntu container and asks it questions with deterministic answers, or gives it jobs like recursively making a directory tree read-only. The database task hands it a real MySQL instance. The knowledge graph task points it at Freebase, which has tens of millions of entities, and grades the answer by F1 because partial credit is the only sensible thing there.

Two are games: a digital card game, and lateral thinking puzzles where the model asks yes/no questions to reconstruct a story. Three were recompiled from existing work: ALFWorld for household tasks, WebShop for shopping, and Mind2Web for web browsing.

The test split is 1,014 problems and the development split is 269. That's deliberately modest, because every problem takes several rounds. The paper estimates five rounds for a database query and thirty-five for a household task, which works out at roughly 11,000 model calls for one full evaluation. AgentBench costs real money to run, and that constraint shaped its size.

## The AgentBench overall score is relative to the paper's model cohort

Here's the part that trips people up. AgentBench's headline figure, which the paper calls OA, is not a percentage and not a success rate. GPT-4 scored 4.01. Nothing is 4.01 out of anything.

The authors had a genuine problem to solve. The eight environments use five different metrics and produce wildly different score distributions: WebShop hands out reward generously, lateral thinking puzzles almost never do. Averaging the raw numbers would have made the benchmark mostly a WebShop benchmark with decorations. So each environment's scores were rescaled so that the average across all the models they tested came to 1, and the overall score is the average of those rescaled values.

That's a reasonable fix with a consequence that is easy to miss when quoting the number. The scale is anchored to the 29 models in the paper's current edition, and those weights were then frozen so later work could reuse them. A model scoring 4.01 is scoring about four times that cohort's average. It's a rank within a historical field, dressed as an absolute measurement, and it cannot be compared across benchmarks or read as "got 4% of something".

For the record, the frozen weights themselves say something. Lateral thinking puzzles got a weight reciprocal of 3.5 and WebShop 30.7, meaning the puzzles were about nine times harder for that cohort than the shopping tasks were.

## Where the AgentBench leaderboard stands

Our compact table omits the source’s per-environment uncertainty columns. Read those alongside the average in the [published FC sheet](https://docs.google.com/spreadsheets/d/e/2PACX-1vRR3Wl7wsCgHpwUw1_eUXW_fptAPLL3FkhnW_rua0O1Ji_GIVrpTjY5LaKAhwO-WeARjnY_KNw0SYNJ/pubhtml) before comparing entries.

| # | Submission | Model org | Overall | Reported by | Dated |
| --- | --- | --- | --- | --- | --- |
| 1 | AgentRL w/ Qwen2.5-32B-Instruct | Z.ai | 70.4 | AgentRL | 10/2025 |
| 2 | AgentRL w/ Qwen2.5-14B-Instruct | Z.ai | 67.7 | AgentRL | 10/2025 |
| 3 | AgentRL w/ GLM-4-9B-0414 | Z.ai | 65 | AgentRL | 10/2025 |
| 4 | AgentRL w/ Qwen2.5-7B-Instruct | Z.ai | 62 | AgentRL | 10/2025 |
| 5 | AgentRL w/ Qwen2.5-3B-Instruct | Z.ai | 60 | AgentRL | 10/2025 |
| 6 | Claude Sonnet 4.5 (2025-09-29) | Anthropic | 58.9 | not stated | 11/2025 |

As of **2026-10-16** · **25** submissions · [AgentBench leaderboard](https://docs.google.com/spreadsheets/d/e/2PACX-1vRR3Wl7wsCgHpwUw1_eUXW_fptAPLL3FkhnW_rua0O1Ji_GIVrpTjY5LaKAhwO-WeARjnY_KNw0SYNJ/pubhtml) · [AgentBench paper](https://arxiv.org/abs/2308.03688)

The published AgentBench FC sheet, exported as CSV and sorted by its own average column. First read on 22 September 2026 and re-read on 12 and 16 October 2026 with no change. The sheet titles itself as updated 18 November 2025 and no row is newer than that. Scores are pass@1 averaged over five containerised environments, not the eight in the paper, and each per-environment figure carries an error bar the table above drops. There is no human baseline for this benchmark and none is shown.

The current leaderboard belongs to AgentBench FC, a function-calling rewrite announced in October 2025. It reports plain pass@1 per environment plus an unweighted average, so unlike the paper's OA these numbers are percentages and mean what they appear to mean.

Look at the column headed "Reported by" before the scores. The sheet calls it Result Source, and 23 of its 25 rows give AgentRL, the maintainers' own framework and paper, which means one team on one harness and is better for comparability than a leaderboard of self-reports. The two Claude Sonnet 4.5 rows, sixth and seventh, give no result source at all.

Then look at the organisation column. The top five entries are all AgentRL models, and [AgentRL](https://github.com/THUDM/AgentRL) is the reinforcement learning framework from the same GitHub organisation that maintains AgentBench; the README says the current function-calling version, announced on 10 October 2025, is integrated with it. The highest-placed model from anywhere else is Claude Sonnet 4.5 in sixth, at 58.9. None of that is improper and none of it is hidden. This organisational relationship is disclosed; benchmark-specific training exposure and its contribution to the ordering are unverified.

## AgentBench shrank from eight environments to five

AgentBench FC covers ALFWorld, the database, the knowledge graph, the operating system and WebShop. The card game, the lateral thinking puzzles and Mind2Web are not in it.

That is a defensible engineering decision. Containerising an environment so anyone can reproduce a run is real work. But it means the abstract's description of a benchmark that "consists of 8 distinct environments", repeated in most descriptions of AgentBench, no longer describes what the live leaderboard measures. Two of the dropped environments were the game-grounded ones, so the benchmark's original claim to breadth is the part that got trimmed.

It also means the paper's numbers and the leaderboard's numbers are two different scales measuring two different task sets. You cannot put GPT-4's 4.01 next to Claude Sonnet 4.5's 58.9 and conclude anything.

## AgentBench has no human baseline

The paper reports no human baseline and the leaderboard shows none.

For some of these environments that's understandable, since asking a person to play a digital card game against a scripted opponent is not obviously informative. For the operating system and database tasks it's a real absence, because those are things people do at work and a competent engineer's score would immediately tell you whether the best operating-system score on the FC sheet, 53.6%, is close to the ceiling or nowhere near it.

The two versions fall short in different ways. The paper's OA is relative to its model cohort, so it can only rank. FC's pass@1 is an absolute task-success rate on five environments, so 70.4% does mean the leader finished about seven tasks in ten. What neither can tell you is whether seven in ten is close to what a careful person would manage, or far below it.

Compare GAIA, which set a human baseline of 92% and was therefore able to announce when it had been passed, or WebArena, whose 78.24% gave three years of submissions something to aim at. AgentBench has a leaderboard with a top. It has no finish line.

## AgentBench, agentbench.app and eth-sri/agentbench are different things

Search for AgentBench and three unrelated projects come back under nearly the same name.

This page is about the first: the academic benchmark from the paper, maintained in the [THUDM/AgentBench repository](https://github.com/THUDM/AgentBench). [agentbench.app](https://www.agentbench.app/) is a separate, independent project that describes itself as a way to "benchmark your Claude Code setup" across 40 tasks in seven domains, scored by rules rather than by a model. [eth-sri/agentbench](https://github.com/eth-sri/agentbench) is a research harness, published under the eth-sri GitHub organisation, for evaluating coding agents under different repository-context settings, which also runs on SWE-bench Lite. A score from one of them says nothing about the others.

## What AgentBench can't tell you about a personal agent

None of the eight environments is your life.

There's no calendar in it, no inbox, no bill to dispute, no contact who needs telling you'll be late. The environments were chosen to be diverse and gradable, and diversity in that sense means unrelated to each other, not representative of anything you'd delegate.

AgentBench does not measure a continuing user relationship; other benchmarks may include seeded history. The model arrives at the database knowing nothing about which tables anyone cares about, and leaves without keeping anything. Memory, standing preferences and anything scheduled are worth zero points across all eight environments, which limits what these environments establish about personal-agent memory.

The framing differs too, in a way worth noticing. AgentBench evaluates models, not products. What ships to you is a model plus a scaffold, a tool set, a memory system and a set of decisions about what it's allowed to touch without asking, and all of that is outside the frame. A model that scores well here tells you the raw material is capable. It tells you nothing about what was built on top of it.

## Where these numbers came from

The leaderboard table is the [published AgentBench FC sheet](https://docs.google.com/spreadsheets/d/e/2PACX-1vRR3Wl7wsCgHpwUw1_eUXW_fptAPLL3FkhnW_rua0O1Ji_GIVrpTjY5LaKAhwO-WeARjnY_KNw0SYNJ/pubhtml) linked from the [project README](https://github.com/THUDM/AgentBench), exported as CSV and sorted by its own average column on 22 September 2026, and re-read unchanged on 12 and 16 October and 5 November 2026: 25 rows, the same top six. The sheet titles itself as updated 18 November 2025 and no row in it is newer, so treat the ranking as almost a year old rather than current.

The paper figures, including the 1,014 test problems, the 269 development problems, the roughly 11,000 inference calls, the eight environments, the 29 models and how OA is computed, come from [the paper itself](https://arxiv.org/abs/2308.03688), in the revision arXiv dates 4 October 2025, and we checked its arXiv page by its own title so we know we read the paper rather than a description of it. The list of environments in the current version comes from the repository README.

One detail that says something about the benchmark's stage of life. The original leaderboard, the one covering all eight environments, is published as an image file in the repository. There is no machine-readable version of it, so any table of those numbers you find elsewhere was transcribed by hand from a screenshot, including the few figures we quote above.

To run AgentBench yourself, start with the [official repository](https://github.com/THUDM/AgentBench) and its AgentBench FC setup instructions. Match the environment version and container setup to the result you intend to compare; the original eight-environment paper and five-environment FC board are different evaluation configurations.

## FAQ

**What is AgentBench?** A benchmark that evaluates large language models as agents across several interactive environments rather than on a single task type. The original version used eight environments and 1,014 test problems; the current function-calling version uses five.

**What does an AgentBench score of 4.01 mean?** It means roughly four times the average of the 29 models in the paper's current edition. The paper's overall score is normalised against that cohort rather than being a percentage, so it can't be compared with scores from other benchmarks or read as a success rate.

**Is there an official AgentBench leaderboard?** Yes, as a published Google spreadsheet linked from the project README, covering the function-calling version. Its own title dates it to November 2025. The original eight-environment leaderboard exists only as an image in the repository.

**Does AgentBench have a human baseline?** No. The paper reports none and the leaderboard shows none. The FC pass@1 rates are absolute task-success rates, so they say how often a model finished, but without a human figure they cannot say how close that is to what a competent person would manage.

**Who made AgentBench?** Researchers from Tsinghua University, Ohio State and UC Berkeley, with Xiao Liu, Hao Yu and Hanchen Zhang as joint first authors. The repository sits under the THUDM organisation on GitHub, which also maintains the FC leaderboard and AgentRL.

**Is agentbench.app the same as AgentBench?** No. agentbench.app is an independent project for benchmarking Claude Code setups on 40 tasks. AgentBench, the subject of this page, is the 2023 academic benchmark from the THUDM repository and the ICLR 2024 paper.

**How does AgentBench compare to WebArena or GAIA?** AgentBench spreads one model across unrelated environments to test general competence. WebArena tests operating a website until the site's state is correct. GAIA tests multi-step research with an exact answer. AgentBench is the broadest and the least grounded in anything you'd personally want done.

**Can a good AgentBench score tell me which assistant to use?** No, and not only because of the cold start. AgentBench scores models, while a personal assistant is a model plus its scaffolding, tools, memory and permissions, and those are where products differ most.

## What a personal agent scorecard has to ask instead

AgentBench answers "how broadly capable is this model when dropped into an unfamiliar interactive environment". That's worth knowing, and it's a property of the raw material rather than of anything you can buy.

The questions about a product are different. Where does the model run and what can it reach on your machine. How does a tool definition get in front of it, and who decided which ones. Where is your memory kept, and can you correct it when it's wrong. What do you get to approve, and how do you reach it. Those questions are on [our scorecard](https://personalagentbench.com/), which publishes completion rates with their intervals and the count of tasks behind them, and no combined score at all.

Read [how we score](https://personalagentbench.com/how-scoring-works/), start with a [teardown](https://personalagentbench.com/agents/), or see [what the other agent benchmarks measure](https://personalagentbench.com/benchmarks/).

## The other benchmark pages

- [**GAIA**Every question arrives cold. Nothing carries over from the last one, so a system that remembers you scores exactly the same as one that doesn't.](https://personalagentbench.com/benchmarks/gaia/)
- [**BrowseComp**Reading, not doing. Nothing is booked, cancelled, paid for or sent.](https://personalagentbench.com/benchmarks/browsecomp/)
- [**WebArena**The sites aren't yours and hold nothing about you. No saved cards, no order history, no account that remembers.](https://personalagentbench.com/benchmarks/webarena/)
- [**OSWorld**Skill, not memory. It asks whether an agent can operate the machine, never whether it knows why you'd want it to.](https://personalagentbench.com/benchmarks/osworld/)
- [**Tau-bench**The user is simulated and the domains are narrow, but this is the closest thing on the list to what a personal agent actually does.](https://personalagentbench.com/benchmarks/tau-bench/)
- [**Terminal-Bench**Almost nothing a personal agent does happens at a terminal.](https://personalagentbench.com/benchmarks/terminal-bench/)
- [**SWE-bench**A closed problem with a written specification. An errand has neither, and it cannot be rolled back.](https://personalagentbench.com/benchmarks/swe-bench/)
- [**AssistantBench**Still one-shot. No standing preferences, no routine that fires next Monday.](https://personalagentbench.com/benchmarks/assistantbench/)

No numbers invented. Personal agents taken apart at the runtime level, on a method built to take more.

[See the scorecard](https://personalagentbench.com/)

---

Published by Personal Agent Bench (https://personalagentbench.com), run by the Zentor team. Zentor builds agent software of its own, so its products are not on the scorecard and will not be added.