GAIA benchmark: what it measures, and why the top score passed the humans

The GAIA benchmark asks questions a person could answer given an afternoon, a browser and some patience. That was the whole idea, and for two years it was the most honest measure of whether an AI assistant could do useful work rather than sound like it could.

• GAIA is 466 questions that take several steps to answer. The answers to 300 of them were never published, which is what keeps its leaderboard from being gamed.
• The paper's own headline was the gap between people and machines: humans scored 92%, GPT-4 with plugins scored 15%.
• As of 21 September 2026 the top submission scores 93.4%. The gap the benchmark existed to measure is gone.
• Every GAIA question arrives with no history, so a system that remembers you scores exactly the same as one that forgets you.
• Nothing in GAIA is an action. Every answer is a fact retrieved, so no run can ever book, cancel, pay or send anything on your behalf.

What the GAIA benchmark actually is

466 questions, built so the hard part is the work rather than the cleverness.

Most of them need several steps. Search for something, open a file, read a table, follow a link, come back with one exact answer. A question might ask you to find a number buried in a PDF that is only linked from a page you have to find first. None of it requires a leap of insight. All of it requires not giving up.

The answers to 300 of those questions were never released. That detail does more work than it looks like it does: a benchmark whose answers are public gets memorised, and a leaderboard built on memorised answers measures nothing at all. Holding the answer key is the only reason the numbers below mean anything.

The design philosophy was a deliberate departure from where benchmarks were heading in 2023. Instead of chasing problems that are hard for people, GAIA went after problems that are easy for people and hard for machines. The reasoning was that a system you would actually trust with your afternoon has to be reliable on the ordinary, not brilliant on the exotic.

Humans scored 92%. GPT-4 with plugins managed 15%.

That 77-point gap was the entire point of the exercise.

How GAIA scores a system, and why it uses three levels

Questions are split into three levels of difficulty, and the split matters more than a single overall percentage suggests.

Level 1 questions need a small number of steps and usually one tool. Level 2 questions chain more steps together and often cross between a search and a file. Level 3 questions are the ones the authors expected to survive: long chains, several tools, and enough branching that a single wrong turn early costs you the answer.

This is why a leaderboard that only shows an overall score hides the interesting part. A system that scores well on Level 1 and collapses on Level 3 is a system that can look things up but cannot hold a plan together. Both of those facts matter, and only one of them shows up in a headline number.

We report Level 3 separately below for exactly that reason.

Where the GAIA leaderboard stands in September 2026

The gap is gone.

Read the top line again. 93.4%, against a human baseline of 92%. Level 3 questions, the ones built to survive, are being answered nine times out of ten.

Look at the dates as well. The top of this leaderboard was set in the first half of 2026 and has barely moved since, which is what saturation looks like from the outside: a cluster of submissions within a point or two of each other, all of them above the number the paper set as its target.

Why a benchmark passing its own human baseline stops being useful

A benchmark is a ruler. It is useful for as long as the things you are measuring fit on it.

When GAIA launched, the best system scored 15% against a human 92%, and every point of improvement told you something real. Today the top ten submissions sit inside a few points of each other and all of them are above the human number. The ruler has run out.

This does not make GAIA a bad benchmark, and it certainly does not mean the systems at the top are as capable as a competent person. It means GAIA has finished the job it was built for. It still answers "is this system broadly competent at multi-step retrieval", which is worth knowing. It no longer answers "which of these should I use", which is the question anyone actually choosing a product is asking.

Benchmarks saturate. That is the normal end of a successful one, and treating a saturated leaderboard as a live ranking is how people end up believing a two-point difference means something.

What the GAIA benchmark cannot tell you about a personal agent

Every question arrives cold.

That one sentence is the whole limitation, and it is structural rather than something a future version could patch. A GAIA run has no yesterday. The system answering question 300 knows nothing it did not know at question 1, so a product that remembers you scores exactly the same as one that forgets you the moment you close the tab.

Think about what that erases. A standing preference is worth zero points. So is a correction you made last week that should have stuck. So is a routine that fires every Monday without being asked, a note about which airline you refuse to fly, a fact about your billing address that should never need asking twice. All of it invisible.

For a research benchmark this is the right call, because holding memory constant is the only way to compare systems fairly. For judging a product you would hand your accounts to, it removes the single feature that separates a personal agent from a search box.

The second gap: nothing in GAIA is an action

Every GAIA answer is a fact retrieved. Nothing is booked, cancelled, disputed, paid for or sent.

That matters because retrieval and action fail in completely different ways. A wrong retrieval gives you a wrong answer, and you notice. A wrong action charges the wrong card, calls the wrong company, sends a message to the wrong person, or cancels something you meant to keep. The cost of being wrong is not symmetrical, and a benchmark made entirely of retrieval never has to price it.

It also means GAIA never tests the parts of a real agent that carry the most risk: what it is allowed to touch, what it asks permission for, what it does when a website changes under it, and what record it leaves behind so you can check afterwards. Those are the questions we built our own scorecard around, because they are the ones with consequences.

How we pulled these numbers, and why that matters

We did not read the figures above off a screenshot.

Every submission in the public results set was pulled through the dataset API, paged until the whole set was in hand, and sorted locally. That returned 3,759 submissions on 21 September 2026. The human baseline, the 466 question count and the 300 withheld answers came from the paper's own abstract, checked against the paper's title so we know we read the right paper rather than a summary of it.

This sounds like pedantry. It is the difference between a number you can defend and a number you copied. Several sites publish GAIA leaderboards whose provenance nobody can check, and at least one competing agent comparison publishes a full grid of scores with no stated method at all. We would rather show a smaller table and tell you exactly where it came from.

One caveat we will state rather than hide. The public results set is self-reported. Anyone can submit, the withheld answers stop outright memorisation, and nothing stops a submission from being an expensive ensemble that nobody would ship. A score near the top says a configuration exists that can answer these questions. It does not say a product you can buy behaves that way, and it says nothing at all about what that configuration cost to run.

The snapshot is stored with its date, its source link and its method. When it goes stale, it will say so.

FAQ

What is the GAIA benchmark? GAIA is a set of 466 questions designed to test whether an AI assistant can complete multi-step work: searching, opening files, following links and returning one exact answer. The answers to 300 of the questions are withheld to keep the leaderboard meaningful.

Who created GAIA and when? It was introduced in a 2023 paper, "GAIA: a benchmark for General AI Assistants", and the public leaderboard has been running since. The paper is the primary source for the question counts and the human baseline quoted on this page.

What is a good GAIA benchmark score? That depends on when you ask. At launch, 15% was the best a leading model could manage against a human baseline of 92%. As of September 2026 the leading submission scores 93.4%, so anything below the high eighties is now well off the pace.

Is GAIA still a useful benchmark? For checking broad competence, yes. For choosing between the systems at the top, no. When the leader has passed the human baseline the benchmark set as its target, the remaining differences are inside the noise.

Does GAIA test whether an agent remembers you? No. Every question is answered without any history, so personalisation, standing preferences and long-term memory earn no points at all. This is the single biggest reason GAIA cannot rank personal agents.

How does GAIA compare to BrowseComp, WebArena or OSWorld? They measure different failures. GAIA is multi-step retrieval with an exact answer. BrowseComp buries a fact and tests whether a system keeps digging for it. WebArena and OSWorld both test doing rather than finding, one inside a browser and one across a whole desktop. None of the four carries any memory between tasks, which is the gap they all share and the reason none of them ranks a personal agent.

Where can I see the GAIA leaderboard? The public results are published as a dataset on Hugging Face, linked from the table above. We pull it through the API rather than reading the rendered page, so the figures here can be reproduced.

The questions a personal agent scorecard has to ask instead

The questions GAIA cannot ask are the ones that decide whether a personal agent is any good. Where does the model actually run. What can it reach on your machine. How does a tool get in front of it. Can you read, or correct, what it believes about you.

Those are the axes on our scorecard, built by taking four of these products apart at the runtime level rather than by scoring them on quiz questions. It carries no numbers yet and says so plainly, because the numbers have to come from recorded task runs, and inventing them would make it exactly the thing this page just spent two sections criticising.

Read how we score, start with the Pine teardown, or see what the other agent benchmarks measure.