BrowseComp: the benchmark humans mostly fail, and what it still can't tell you

BrowseComp is the mirror image of every other agent benchmark on this list. Where GAIA asked questions a person could answer in an afternoon, BrowseComp asks questions the people who built it mostly could not answer at all.

• 1,266 questions, each one a fact that is genuinely hard to find and easy to check once you have it.
• The people paid to solve them managed 29.2%. On seven questions out of ten they gave up.
• A browsing agent trained for the job scored 51.5% at launch. A frontier model without browsing scored 0.6%.
• There is no continuously updated public leaderboard, so every BrowseComp figure you see quoted comes from a launch announcement or from someone's own report.
• Nothing in it is an action, and nothing in it carries over between questions, which is why it cannot rank a personal agent.

What BrowseComp actually is

1,266 questions, each pointing at a fact buried somewhere on the open web.

The design constraint is the interesting part. Every question has to be hard to find and easy to verify. A question whose answer is a paragraph of judgement is useless here, because you could argue about whether the answer was right. A question whose answer is one specific name, number or date is perfect, because once you have it there is nothing left to debate.

That inversion is what makes the benchmark work. You cannot game it by writing plausible prose. Either you found the thing or you did not.

The questions were built by trying to fail

Most benchmarks are built by writing questions and then checking whether models can answer them. BrowseComp was built the other way around, and the process is worth describing because it explains the numbers.

Trainers wrote tasks with a rule attached: the task had to be one that another person would struggle with. Anything solved more than 40% of the time by other trainers was sent back to be made harder. On top of that, each question had to survive a check against the models of the day, including a frontier model with and without browsing, a reasoning model, and an early version of a browsing agent. If any of them cracked it, the question did not make the cut.

So the dataset is not a sample of hard questions. It is a filtered set of questions that had already defeated both people and machines before they were allowed in.

Where the numbers stand, and why there is no live leaderboard

Read the spread rather than the top line. A frontier model with no browsing scores 0.6%, which is almost exactly zero, and adding browsing takes it to 1.9%. A reasoning model reaches 9.9%. An agent trained specifically to keep browsing until it finds the thing reaches 51.5%.

That is a fifty-point gap created by persistence rather than by intelligence. The models at the bottom of that table are not stupid; they are impatient. They search once, do not find it, and answer anyway.

The other thing to notice is what is missing. GAIA publishes a leaderboard that anyone can submit to, which is why our GAIA page can show you a table pulled through an API and dated to the day. BrowseComp has no such thing. The figures above come from the benchmark author's own launch evaluation, which means two things: they are from the launch rather than from today, and every other BrowseComp number you encounter is somebody's self-report with no shared scoring harness behind it.

We would rather tell you that than dress a launch table up as a live ranking.

A human baseline of 29.2% is the whole story

When the benchmark was verified, trainers were given the questions and a time limit. They solved 367 of 1,255. The other 888, more than seven in ten, they could not solve at all.

Where they did solve a question, they agreed with the reference answer 86.4% of the time, which tells you the questions are well posed rather than ambiguous. The failures are failures of finding, not failures of understanding.

Set that beside GAIA, where the human baseline was 92% and the leading system now scores 93.4%. Two benchmarks, both measuring browsing agents, with human baselines sixty points apart. They are not measuring the same thing at all. GAIA measures whether a system can carry out a multi-step errand competently. BrowseComp measures whether it will keep going when the first four searches turn up nothing.

Both matter. Neither is the whole picture, and quoting one without the other is how a product ends up looking better than it is.

Why "easy to verify" is the design decision that matters

Every question on BrowseComp has an answer you can check in a second once somebody hands it to you. That constraint sounds like a convenience. It is actually the thing holding the whole benchmark up.

Consider the alternative. A benchmark that asks for an assessment, a summary or a recommendation needs a judge, and a judge is either a person, which does not scale, or another model, which ends up measuring the agreement between two models rather than the quality of one. Every benchmark that has tried to score open-ended output has run into this, and most of them quietly became measurements of how much the grader liked the writing style.

By insisting on answers that are one name, one number or one date, BrowseComp removed the judge entirely. Scoring is a string comparison. Nobody argues about it, nobody can lobby for a better read of their output, and a result from last year is comparable with a result from today.

The cost of that choice is scope. Anything a personal agent does that cannot be reduced to one verifiable token is outside what this benchmark can see, which covers most of the work you would actually delegate: writing the email, deciding whether the refund offer is good enough, judging whether to escalate. The benchmark is rigorous precisely because it declined to look at those.

What repeated attempts buy, and what that implies about cost

One finding in the launch evaluation deserves more attention than it gets: letting the system try a question several times and then pick among its answers improves accuracy by 15% to 25% over a single attempt.

That is a large gain, and it is bought entirely with compute. Nothing about the model changed. It simply got more chances.

For a benchmark leaderboard this is an interesting result about sampling. For anyone choosing a product it is a warning: a headline score can reflect a configuration that runs the same question many times over, and nobody ships that configuration to a consumer product at a consumer price. A number near the top of a table describes what was possible in an evaluation, not what happens when you ask your assistant something on a Tuesday.

What BrowseComp cannot tell you about a personal agent

Reading, not doing.

Every BrowseComp answer is a fact retrieved. Nothing is booked, cancelled, disputed, paid for or sent, and no wrong answer costs you anything except a wrong answer. The entire class of failures that makes a personal agent frightening, which is acting confidently on your behalf and getting it wrong, is outside the scope of the test.

And as with every other benchmark on this bench, each question arrives cold. There is no yesterday. A system that remembers you refuse red-eye flights, that has your account numbers, that learned last month which of your three email addresses is the real one, scores exactly the same as a system that knows nothing about you. Memory is worth zero points.

There is a third gap that is specific to this benchmark. BrowseComp rewards persistence, and persistence is not free. An agent that searches thirty times to find one fact is expensive, slow, and in a consumer product would be interrupted long before it finished. The benchmark measures a behaviour that is correct in a lab and often impractical in a product, and it has no way to tell you which one you are looking at.

So what should you measure instead

If persistence and retrieval are not the questions, the questions are these. Where does the model run, and what can it reach on your machine. How does a tool get in front of it, and who decided which ones. Where does your memory live, and can you correct it. How do you reach the thing, and what can start it without you.

Those are the five axes on our scorecard, built by taking four personal agents apart at the runtime level. It carries no numbers yet, and it says so: numbers have to come from recorded task runs, and inventing them would make us exactly the sort of comparison this page has spent two sections warning you about.

FAQ

What is BrowseComp? A benchmark of 1,266 questions, each asking for a fact that is hard to find on the open web and simple to verify once found. It was built to measure whether a browsing agent keeps searching when the answer does not turn up quickly.

How hard is BrowseComp for humans? Very. In the verification campaign, trainers solved 367 of 1,255 problems, which is 29.2%. More than seven in ten were left unsolved. Where they did answer, they matched the reference answer 86.4% of the time.

What is the best BrowseComp score? At launch, a browsing agent trained for the task scored 51.5%, against 9.9% for a reasoning model, 1.9% for a frontier model with browsing enabled, and 0.6% without. There is no live leaderboard, so newer figures are self-reported rather than measured on a shared harness.

Is there an official BrowseComp leaderboard? No. Unlike GAIA, which publishes results anyone can submit to, BrowseComp's public numbers come from the launch evaluation. Any ranking you find elsewhere is somebody's compilation, and it is worth asking where each row came from.

How is BrowseComp different from GAIA? GAIA has a human baseline of 92% and is effectively saturated, with the top system above that baseline. BrowseComp has a human baseline of 29.2% and remains far from solved. GAIA asks whether a system can complete an errand. BrowseComp asks whether it will keep looking.

Does BrowseComp measure whether an agent can act on my behalf? No. Every answer is a fact retrieved. Nothing is booked, cancelled, sent or paid for, so the failure mode that matters most in a personal agent, acting wrongly with confidence, is not tested at all.

Why a browsing score is not a personal agent score

BrowseComp is a good benchmark, built carefully, measuring something real that other benchmarks miss. A 51.5% against a human 29.2% is a genuine result and worth respecting.

It still tells you nothing about whether a product should be trusted with your accounts, because it never asks a system to remember you, act for you, or explain afterwards what it did. Those questions have their own page: how we score, the four teardowns, and what the other benchmarks measure.