WebArena: the benchmark that rebuilt the web so the test would hold still

The WebArena benchmark solves a problem most web agent evaluations pretend doesn't exist. If you score an agent against a real website, you're scoring it against a thing that changed while you were measuring, and next month's run isn't comparable to this month's. So the authors built their own web instead.

• 812 tasks across four kinds of site the paper names, e-commerce, a social forum, collaborative software development and content management, with a map, a wiki, a calculator and a scratchpad alongside them as tools.
• Success is checked programmatically against the end state of the site, not against text the agent wrote about what it did.
• The original paper's best GPT-4 agent scored 14.41% against a human baseline of 78.24%. The leaderboard's top entry now scores 74.3%.
• That human baseline was measured on one sampled task per template by five graduate students, not on the 812 tasks the agents run. The two numbers are close together and are not the same measurement.

What the WebArena benchmark actually is

A set of working websites you host yourself, plus 812 tasks phrased the way a person would phrase them.

The sites are functional rather than mocked. There's an e-commerce shop with a product catalogue, orders and customer reviews; the seller-side admin interface behind it; a GitLab instance with repositories, issues and commit history; a forum modelled on Reddit; and a map. Around those sit a wiki, a calculator and a scratchpad, so a task can require looking something up or holding a number while doing something else.

The tasks read like requests rather than instructions. Cancel order 307. Tell me the number of reviews mentioning the word "best". Find the closest national park to the largest city in Maine. Add a new colour option to one size of one product. A few need two sites at once, which is why the environment supports multiple tabs, and seventeen of them pair the wiki with the map.

Counted from the repository's own task file on 22 September 2026, the distribution runs 187 tasks on the shop, 182 on its admin side, 180 on GitLab, 109 on the map and 106 on the forum, with the remainder spanning two sites.

Why self-hosting is the whole design

Everything WebArena is good at follows from the decision to ship the websites rather than point at somebody else's.

A live site changes its layout, its stock, its prices and its data, so two agents evaluated a month apart were not given the same test. Worse, a live site can be modified by the agent in ways that affect the next agent's run. Self-hosted and resettable means a result from 2023 is comparable with a result from 2026, which is a rare property and the reason this leaderboard is still worth reading three years in.

The second consequence is how grading works. Because the site is yours, you can inspect it afterwards. WebArena checks functional correctness: did the order actually get cancelled, does the repository actually exist, is the returned string the right one. An agent that narrates a successful cancellation without cancelling anything scores zero, which is exactly the failure a text-similarity grader would miss.

Where the WebArena leaderboard stands

The shape of that table matters more than the top line. When the paper was published, the best GPT-4 agent reached 14.41% and the authors' point was the size of the gap. The leading entry now sits at 74.3%, which means roughly sixty points have been closed on a benchmark whose environment has not moved.

Two details are worth pulling out of the sheet. The leaderboard records, per row, whether the submission released its code and trajectories, and two of the entries near the top are marked self-reported with nothing behind them. And the highest-scoring entries are not frontier models being run as-is; they're scaffolds, memory schemes and fine-tunes built specifically for this environment. A row here tells you what a configuration achieved, not what a product does.

The sheet is still being added to. Its newest row is dated June 2026, which puts WebArena in an unusual position for a 2023 benchmark: alive, unsaturated, and with about four points of headroom left before it runs into the human number.

The human baseline is not measured on the same tasks

That 78.24% gets quoted constantly, including by us, and almost nobody quoting it says how it was produced.

Five computer science graduate students were each given one sampled task from each of the 170 templates the tasks were generated from. They completed 78.24% of them, taking an average of 110 seconds per task, and did better on tasks that required changing something (81.32%) than on tasks that required finding something out (74.68%). The failures were mostly misreadings: answering with travel distance when asked for travel time, giving a name when asked for a name and an email, filling in half of a product's details.

So the human figure and the agent figures are not the same measurement. One is five people on roughly 170 sampled tasks; the other is a system on all 812. The leaderboard sheet flags this in a note that survives almost no citation. Treat the gap as a useful signal about difficulty and not as a like-for-like scoreline, and be sceptical of anyone announcing that agents have passed humans on WebArena without mentioning any of the above.

Unachievable tasks, and the trick they catch

Some WebArena tasks cannot be completed, on purpose. The information isn't there, the permission isn't there, or the site simply doesn't do that.

This is the most quietly clever thing in the benchmark, because it tests the behaviour that makes an agent safe to leave alone: knowing when to stop. An agent that confidently reports success on an impossible task is worse than useless, since you'd have no reason to check.

The paper's numbers on this are striking. Told explicitly to stop when a task looks impossible, GPT-4 correctly gave up on 77.78% of the unachievable set but its overall score dropped, because the same caution made it abandon achievable tasks too. Without the hint it scored higher overall and identified only 44.44% of the impossible ones. The humans got 100% of them.

That trade-off has not gone away and it is not a prompting problem. Caution and completion pull against each other, and every product on our scorecard has made a call about where to sit on that line without telling you which call it made.

What WebArena can't tell you about a personal agent

The sites aren't yours, and they hold nothing about you.

That's the structural limit, and it isn't something a future version could patch without becoming a different benchmark. The shop has no saved card of yours, no delivery address you've used twice, no history of the three things you always reorder. The GitLab instance doesn't know which repositories you care about. Nothing an agent learns in task 200 is available to it in task 201, so a product that remembers you scores exactly what a product that forgets you scores.

Consider what that removes. A standing preference is worth nothing. A correction you made last week is worth nothing. A routine that fires every Monday without being asked cannot be expressed as a WebArena task at all, because there's nobody for the agent to have a relationship with.

There's a second gap, subtler than the first. Every WebArena task is initiated by a person typing an instruction. Nothing in the benchmark starts because a clock reached a time or an email arrived, and the products we've taken apart mostly can start that way. The whole question of what an agent does when a stranger addresses it is outside the frame.

And a third, which is about consequence. WebArena's shop takes no real money. An agent that cancels the wrong order in a Docker container has caused no harm, so nothing in the score reflects the asymmetry between a wrong answer and a wrong action. That asymmetry is most of what makes a personal agent worth being careful about.

Where these numbers came from

We didn't read any of the figures above off a screenshot or a summary.

The 812 task count is a count: the repository's own task configuration file, fetched and counted on 22 September 2026, which also gives the per-site distribution quoted earlier. The leaderboard table is the sheet the project page links to, exported as CSV and sorted locally, with the self-reported flags taken from the sheet's own columns. The paper figures, including the human baseline and the unachievable-task results, come from the paper, and we checked its arXiv page by its own title so we know we read the paper rather than something describing it.

One thing to flag if you go looking. The project site has become WebArena-x, a suite covering several benchmarks, and the original WebArena project page now lives one level down at /og/. That page still carries the leaderboard link, which is the only place the current numbers exist.

We'd rather show a five-row table and tell you exactly where each column came from than a larger one you have to take on trust.

FAQ

What is the WebArena benchmark? A self-hosted web environment with 812 tasks across a shop, an admin back office, a GitLab instance, a forum and a map. Agents are graded on whether the site ended up in the right state, not on what they said they did.

What is a good WebArena score? As of September 2026 the leading entry on the public leaderboard scores 74.3%, against a human figure of 78.24% measured on a sampled subset. The original paper's best GPT-4 agent managed 14.41%, so anything in the teens now reflects an unassisted model rather than a serious attempt.

Is WebArena still maintained? The leaderboard sheet is still receiving submissions, with entries dated into mid-2026, and the environment itself is unchanged since release. That stability is the point: results from different years remain comparable.

How is WebArena different from GAIA? GAIA asks multi-step research questions and grades the answer string. WebArena asks an agent to change something on a website and grades the website afterwards. GAIA is saturated, with its leader above the human baseline; WebArena is not.

Does WebArena test memory or personalisation? No. Every task starts from a reset environment that holds nothing about the agent or any user, so long-term memory, standing preferences and scheduled routines earn no points at all.

Can I run WebArena myself? Yes, which is unusual and is the reason its results are reproducible. The sites ship as Docker images and the task set is a configuration file in the public repository, so a published score can be checked rather than believed.

The questions a personal agent scorecard has to ask instead

WebArena answers "can this system operate a website it has never seen, reliably enough to be graded on the result". That's a real question, answered rigorously, and it isn't the question you're asking when you decide whether to give something your email and a payment method.

Those questions are on our scorecard: where the model runs and what it can reach on your machine, how a tool definition gets in front of it, where your memory is kept and whether you can correct it, and what can start the agent working while you're asleep. It publishes completion rates from the first task runs with their intervals and sample sizes attached, marks the highest one unusable because 21 tasks can't hold it, and refuses to fold any of it into a single number.

Read how we score, start with a teardown, or see what the other agent benchmarks measure.