GAIA benchmark: leaderboard scores and their limits

The GAIA benchmark is a set of 466 multi-step questions that a person could answer with a browser, some patience and usually well under half an hour. That was the whole idea, and by 2 November 2026 its public leaderboard had drawn 4,079 submissions from teams testing whether an AI assistant can do useful work rather than sound like it could.

• GAIA is 466 questions that take several steps to answer, from Meta, Hugging Face and AutoGPT researchers. The answers to 300 of them were never published, which reduces direct answer leakage but does not rule out contamination or repeated leaderboard tuning.
• The paper's own headline was the gap between people and machines: humans scored 92%, GPT-4 with plugins scored 15%.
• As of 2 November 2026 the top self-submitted entry scores 94.02% on the test set. Princeton's HAL, which reruns agents itself on the public validation set, has a best of 74.55%.
• Every GAIA question arrives with no history, so a system that remembers you scores exactly the same as one that forgets you.
• Nothing in GAIA is an action. Every answer is a fact retrieved, so no run can ever book, cancel, pay or send anything on your behalf.

What the GAIA benchmark actually is

466 questions, built so the hard part is the work rather than the cleverness.

Most of them need several steps. Search for something, open a file, read a table, follow a link, come back with one exact answer. The Level 1 sample in the paper's first figure asks for the actual enrolment count of a 2018 clinical trial on H. pylori in acne patients, as listed on the NIH website. None of it requires a leap of insight. All of it requires not giving up. The paper puts human annotators at between 6 minutes for the simplest questions and 17 minutes for the hardest.

The paper says the authors released a developer set of 166 annotated questions and kept the answers to the remaining 300, which are the test set the leaderboard scores. Later descriptions, including Princeton's HAL leaderboard and the Berkeley exploit write-up below, give the public validation set as 165 questions, so check the dataset version before quoting an exact split. The dataset is gated on Hugging Face: you accept the terms, download it, run your agent however you like and upload the answers. The maintainers ask that the validation and test questions not be reshared in crawlable form, which is why this page paraphrases the paper's own published sample rather than any dataset question.

That split does more work than it looks like it does. A benchmark whose answers are public gets memorised, and in April 2026 a Berkeley team showed exactly that: an exploit agent that read the public validation answers from a file, plus a quirk in the answer normaliser, scored about 98% on the validation questions without solving any. Holding the test answer key is the main reason the numbers below mean anything.

The design philosophy was a deliberate departure from where benchmarks were heading in 2023. Instead of chasing problems that are hard for people, GAIA went after problems that are easy for people and hard for machines. The reasoning was that a system you would actually trust with your errands has to be reliable on the ordinary, not brilliant on the exotic.

Humans scored 92%. GPT-4 with plugins managed 15%.

That 77-point gap was the entire point of the exercise.

How GAIA scores a system, and why it uses three levels

Questions are split into three levels of difficulty, and the split matters more than a single overall percentage suggests.

The paper's working definitions are loose on purpose. Level 1 questions generally need no tools, or at most one tool and no more than five steps. Level 2 questions take roughly five to ten steps and need different tools combined. Level 3 questions are for a near-perfect general assistant: arbitrarily long sequences of actions, any number of tools, and access to the world in general.

Scoring is simple. Every answer is a short string, a number or a comma-separated list, and it is graded by quasi exact match against the single ground truth, after normalisation that depends on the answer type. There is no model judge. That makes the score cheap to compute and easy to audit, and it is also why a formatting quirk in the normaliser could be exploited, as the Berkeley work showed.

This is why a leaderboard that only shows an overall score hides the interesting part. A system that scores well on Level 1 and collapses on Level 3 is a system that can look things up but cannot hold a plan together. Both of those facts matter, and only one of them shows up in a headline number.

We report Level 3 separately below for exactly that reason.

Where the GAIA leaderboard stands now

The gap is gone.

Read the top line again. 94.02%, against a human baseline of 92%. Level 3 questions, the ones built to survive, are being answered by the first-listed system more than nine times out of ten.

Look at the dates as well. Submissions keep arriving: 53 more landed between 13 and 16 October without changing the top five, and more than a hundred have arrived since. The leader changed twice in three weeks, on 22 September and again on 12 October, and both times the new first place was about a third of a point above the one it displaced. On 29 October KeenData's Keen AgenticOS V1.0 tied it at 94.02, and the next day OmniSeek_Preview_v1.1, from Alibaba and HKUST (Guangzhou), entered the top five at 93.36. One team, Shadow, submitted 13 variants of its system between 8 and 12 October, and one of them took first place. In the full set on 30 October, 51 submissions sit above the 92% human figure. That is what saturation looks like from the outside. Not a board that stops moving, but one where everything still moving is inside a single point, all of it above the number the paper set as its target.

Two submissions tie for first at 94.02 and two more tie at 93.36 in rows four and five. Within each tie the earlier submission is listed first, which is not evidence of a performance difference. Below them, 11 of the top 20 tie at 93.02.

Why a benchmark passing its own human baseline stops being useful

A benchmark is a ruler. It is useful for as long as the things you are measuring fit on it.

When GAIA launched, the best system scored 15% against a human 92%, and every point of improvement told you something real. Today every submission in the table above clears the human number, and the top five sit within about a point of each other. The ruler has run out.

Independent reruns tell a less dramatic story. Princeton's Holistic Agent Leaderboard runs agents itself on the 165 public validation questions and marks each row as reproduced by its team. Read on 16 October 2026, its best entry was its own generalist agent on Claude Sonnet 4.5 at 74.55%, with a reported cost of about $178 for the run. HAL says it has paused adding new models and that its cost figures are overstated by up to a factor of two for some rows. It is a different split and a different setup from the test board, so the two numbers are not directly comparable, but they are a reminder that a 94% self-submission and a 75% third-party rerun can both be true at once.

This does not make GAIA a bad benchmark, and it certainly does not mean the systems at the top are as capable as a competent person. It means GAIA has finished the job it was built for. It still answers "is this system broadly competent at multi-step retrieval", which is worth knowing. It no longer answers "which of these should I use", which is the question anyone actually choosing a product is asking.

Benchmarks saturate. That is the normal end of a successful one, and treating a saturated leaderboard as a live ranking is how people end up believing a two-point difference means something.

What the GAIA benchmark cannot tell you about a personal agent

Every question arrives cold.

That one sentence is the whole limitation, and it is structural rather than something a future version could patch. A GAIA run has no yesterday. The system answering question 300 knows nothing it did not know at question 1, so a product that remembers you scores exactly the same as one that forgets you the moment you close the tab.

Think about what that erases. A standing preference is worth zero points. So is a correction you made last week that should have stuck. So is a routine that fires every Monday without being asked, a note about which airline you refuse to fly, a fact about your billing address that should never need asking twice. All of it invisible.

For a research benchmark this is the right call, because holding memory constant is the only way to compare systems fairly. For judging a product you would hand your accounts to, it removes the single feature that separates a personal agent from a search box.

The second gap: nothing in GAIA is an action

Every GAIA answer is a fact retrieved. Nothing is booked, cancelled, disputed, paid for or sent. (Gaia2, covered in the FAQ, was built partly to fix this.)

That matters because retrieval and action fail in completely different ways. A wrong retrieval gives you a wrong answer, and you notice. A wrong action charges the wrong card, calls the wrong company, sends a message to the wrong person, or cancels something you meant to keep. The cost of being wrong is not symmetrical, and a benchmark made entirely of retrieval never has to price it.

It also means GAIA never tests the parts of a real agent that carry the most risk: what it is allowed to touch, what it asks permission for, what it does when a website changes under it, and what record it leaves behind so you can check afterwards. Those are the questions we built our own scorecard around, because they are the ones with consequences.

How we pulled these numbers, and why that matters

We did not read the figures above off a screenshot.

Every submission in the public results set was pulled from the dataset's parquet export on Hugging Face and sorted locally. That returned 4,079 submissions on 2 November 2026, against 3,759 when this page was first written in September 2026 and 3,896 on 12 October. The human baseline, the 466 question count and the 300 withheld answers came from the paper's own abstract, and the 166-question developer set, the level definitions and the scoring rule from its body, checked against the paper's title so we know we read the right paper rather than a summary of it.

This sounds like pedantry. It is the difference between a number you can defend and a number you copied. Several sites publish GAIA leaderboards whose provenance nobody can check. We would rather show a smaller table and tell you exactly where it came from.

One caveat we will state rather than hide. The public board accepts system submissions. Withholding test answers limits direct access to the answer key but does not rule out contamination or memorisation. A submitted configuration may also differ from a shipping product; we have not verified each entry's execution or cost. A score near the top says a configuration exists that can answer these questions. It does not say a product you can buy behaves that way, and it says nothing at all about what that configuration cost to run.

The snapshot is stored with its date, its source link and its method. When it goes stale, it will say so.

FAQ

What is the GAIA benchmark? GAIA is a set of 466 questions designed to test whether an AI assistant can complete multi-step work: searching, opening files, following links and returning one exact answer. The answers to 300 of the questions are withheld to keep the leaderboard meaningful.

How is GAIA scored? By quasi exact match: each answer is a short string, number or list, normalised by type and compared with the single ground truth. The leaderboard reports overall accuracy on the 300 test questions plus a score for each of the three levels.

Who created GAIA and when? Researchers from FAIR and GenAI at Meta, Hugging Face and AutoGPT, led by Grégoire Mialon, in the paper "GAIA: a benchmark for General AI Assistants", first posted in November 2023 and published at ICLR 2024. The affiliations are as listed on the paper's Hugging Face page. The paper is the primary source for the question counts and the human baseline quoted on this page, and the public leaderboard has been running since.

What is a good GAIA benchmark score? That depends on when you ask. At launch, 15% was the best a leading model could manage against a human baseline of 92%. As of 2 November 2026 the leading self-submitted entry scores 94.02% on the test set, while the best independent rerun on Princeton's HAL, on the validation set, is 74.55%. Which one counts as good depends on whether you trust the submitter's own setup.

Is GAIA still a useful benchmark? For checking broad competence, yes. For choosing between the systems at the top, no. When the leader has passed the human baseline the benchmark set as its target, small remaining differences need uncertainty and paired-task analysis before being treated as meaningful.

Does GAIA test whether an agent remembers you? No. Every question is answered without any history, so personalisation, standing preferences and long-term memory earn no points at all. This is the single biggest reason GAIA cannot rank personal agents.

What is Gaia2? GAIA's successor, released by Meta and Hugging Face on 22 September 2025 with an open framework called ARE. Where GAIA was read-only, Gaia2 runs agents in a mock smartphone with email, calendar, contacts and shopping apps, and checks the actions they take, so it does test doing as well as finding. Its Gaia2 paper (February 2026) reports GPT-5 (high) as the best model at 42% pass@1. The world is still simulated and the subjects are models, not consumer products.

How does GAIA compare to BrowseComp, WebArena or OSWorld? They measure different failures. GAIA is multi-step retrieval with an exact answer. BrowseComp buries a fact and tests whether a system keeps digging for it. WebArena and OSWorld both test doing rather than finding, one inside a browser and one across a whole desktop. OSWorld 2.0 gives some tasks seeded user history, but none of the four measures what an agent learns about a person across a continuing relationship, which is the gap they share and the reason none of them ranks a personal agent.

Where can I see the GAIA leaderboard? The public results are published as a dataset on Hugging Face, also linked from the table above. We pull it through the API rather than reading the rendered page, so the figures here can be reproduced.

The questions a personal agent scorecard has to ask instead

The questions GAIA cannot ask are the ones that decide whether a personal agent is any good. Where does the model actually run. What can it reach on your machine. How does a tool get in front of it. Can you read, or correct, what it believes about you.

Those questions sit on our scorecard, built by taking four personal agents apart at the runtime level rather than by scoring them on quiz questions. It carries completion rates from task runs in adapted product runtimes with a substituted exam model, each with a 95% interval and the number of tasks behind it, and no combined score, because a single number would make it exactly the thing this page just spent two sections criticising.

Read how we score, start with the Pine teardown, or see what the other agent benchmarks measure.

The other benchmark pages

• BrowseCompReading, not doing. Nothing is booked, cancelled, paid for or sent.
• WebArenaThe sites aren't yours and hold nothing about you. No saved cards, no order history, no account that remembers.
• OSWorldSkill, not memory. It asks whether an agent can operate the machine, never whether it knows why you'd want it to.
• Tau-benchThe user is simulated and the domains are narrow, but this is the closest thing on the list to what a personal agent actually does.
• AgentBenchNone of the environments is your life. No calendar, no inbox, no bill to dispute.
• Terminal-BenchAlmost nothing a personal agent does happens at a terminal.
• SWE-benchA closed problem with a written specification. An errand has neither, and it cannot be rolled back.
• AssistantBenchStill one-shot. No standing preferences, no routine that fires next Monday.