BrowseComp benchmark: hard lookups and leaderboard scores

BrowseComp is a benchmark of 1,266 hard-to-find questions for browsing agents, published by OpenAI in April 2025 with a paper, and graded by a model against short reference answers. Where GAIA asked questions a person could usually answer in well under half an hour, BrowseComp asks questions that the trainers who checked it mostly could not answer in two hours.

• In OpenAI's verification campaign, human trainers solved 29.2% and gave up on 70.8% after trying for two hours. That is a solve rate under those rules, not a ceiling.
• At launch, OpenAI's Deep Research agent scored 51.5% and GPT-4o without browsing 0.6%.
• The one independently run board we found is Kaggle's: 42 models, last updated March 2026, led by Gemini 3.1 Pro Preview at 31.2% when we re-read it on 16 October 2026.
• Vendor model cards now report scores above 90%, each on the vendor's own setup rather than a shared harness.
• Nothing in it is an action, and nothing in it carries over between questions, which is why it cannot rank a personal agent.

What BrowseComp actually is

1,266 questions, each pointing at a fact buried somewhere on the open web.

The design constraint is the interesting part. Every question has to be hard to find and easy to verify. A question whose answer is a paragraph of judgement is useless here, because you could argue about whether the answer was right. A question whose answer is one specific name, number or date is perfect, because once you have it there is nothing left to debate.

The paper's own example shows the shape. It asks for the title of an EMNLP paper from 2018 to 2023 whose first author did their undergraduate degree at Dartmouth and whose fourth author did theirs at Penn. Checking a proposed answer takes a few searches. Finding it by brute force means reading the author lists of thousands of papers. Trainers were told to start from a fact and work backwards in exactly that way.

That inversion is what makes the benchmark work. You cannot game it by writing plausible prose. Either you found the thing or you did not.

How the BrowseComp questions were built: by trying to fail

Most benchmarks are built by writing questions and then checking whether models can answer them. BrowseComp was built the other way around, and the process is worth describing because it explains the numbers.

According to the paper, trainers had three checks. They had to confirm that GPT-4o with and without browsing, o1 and an early version of Deep Research could not answer the question. They had to run five simple Google searches and confirm the answer was not on the first page of any of them. And the question had to be one another person would not solve within ten minutes. That last rule was not strictly enforced: a second trainer attempted only a portion of the questions, and trainers whose questions were solved more than 40% of the time were asked to revise them.

So the dataset is not a random sample of hard questions. It is a set screened against the models of early 2025 before it was allowed in, which also means no later model was ever screened against it.

Where BrowseComp scores stand: the Kaggle leaderboard and the launch figures

That table is Kaggle's BrowseComp board, listed under OpenAI's organisation on Kaggle Benchmarks and marked as maintained by Kaggle. Each row is a model version run once, on Kaggle's own setup, with an interval printed beside the score. It held 42 models when we first read it on 12 October 2026 and again, unchanged, on 16 October 2026; the page header gives its last update as 26 March 2026, so it is a maintained board rather than a live one. It also does not say what tools each model was given, which matters: OpenAI's own launch run put o1 at 9.9%, and Kaggle's board puts o1 at 6.3%. Same questions, different harness, different number.

The launch evaluation is still the most instructive table, because it compares setups rather than models. Read the spread rather than the top line. A frontier model with no browsing scores 0.6%, which is almost exactly zero, and adding browsing takes it to 1.9%. A reasoning model reaches 9.9%. An agent trained specifically to keep browsing until it finds the thing reaches 51.5%.

That is a roughly fifty-point gap between the evaluated systems. Persistence may contribute, but the table does not isolate it from model, tooling or budget differences. It reports what these configurations achieved, not a controlled estimate of why one did better.

The other thing to notice is what neither table is. GAIA publishes a leaderboard that anyone can submit to, which is why our GAIA page can show a table pulled through an API and dated to the day. We found no open submission route for BrowseComp. The Kaggle rows are Kaggle's runs, the launch rows are OpenAI's, and most other BrowseComp numbers you meet, including the ones in model announcements and on aggregator sites, are self-reports with no shared harness behind them.

Those self-reports are now far above both tables. The Kimi K3 model card that Moonshot AI published on Hugging Face, read on 16 October 2026, reports 91.2% for Kimi K3 with context compaction triggered at 300K tokens, and 90.4% with the full context window and no compaction. The same table lists 90.4% for GPT-5.6 Sol and 88.0% for Claude Fable 5, figures the card says it cited from OpenAI and Anthropic. We have not checked those two against OpenAI's or Anthropic's own pages, and none of the three ran on Kaggle's setup. Set beside Kaggle's 31.2%, they show how much of a BrowseComp score belongs to the harness, the context budget and the compute rather than to the model name.

Variants add to the confusion. BrowseComp-Plus (August 2025) fixes the document collection instead of searching the live web, so retrievers can be compared fairly; its paper reports GPT-5 at 55.9% with BM25 retrieval and 70.1% with Qwen3-Embedding-8B retrieval, a figure that belongs to that fixed corpus and not to the original test. BrowseComp-ZH is 289 questions built the same way for the Chinese web. A score with "BrowseComp" in its name is only comparable with another score on the same variant and setup.

What BrowseComp's 29.2% human figure measures

In the verification campaign, trainers attempted 1,255 of the questions; the paper says eleven of the 1,266 were left out for various reasons. They could give up only after working on a question for about two hours. They solved 367, which is 29.2%, and gave up on the other 888.

Of the 367 they solved, 317 matched the reference answer, which is 86.4%. That conditional agreement does not establish that every question is unambiguous, and the paper itself says questions built backwards from a fact are likely but not guaranteed to have only one valid answer. It also suggests that experienced professionals, detectives for example, could solve some of the ones the trainers gave up on. So 29.2% is what generalist trainers managed under a two-hour rule, not the most a person could do.

Set that beside GAIA, where the human figure was 92% and the leading system scored 94.02% as of 2 November 2026. Two benchmarks, both measuring browsing agents, with human figures sixty points apart. They are not measuring the same thing at all. GAIA measures whether a system can carry a multi-step lookup through to one exact answer. BrowseComp measures whether it will keep going when the first four searches turn up nothing.

Both matter. Neither is the whole picture, and quoting one without the other is how a product ends up looking better than it is.

Why "easy to verify" is the design decision that matters

Every question on BrowseComp has an answer you can check in a second once somebody hands it to you. That constraint sounds like a convenience. It is actually the thing holding the whole benchmark up.

Short reference answers narrow the grading question compared with open-ended assessments. They do not eliminate judgement: the official grader still decides whether the extracted response has meaningful differences from the reference. That makes the grading model and prompt part of the evaluation setup, alongside the browsing agent and its search budget.

BrowseComp uses short reference answers, but the official evaluator calls a grader model to extract the final answer and assess whether it matches the reference. It is not plain string comparison. Comparisons across dates still require the same dataset variant, tool access, model, budget and grading setup; a short answer alone does not make configurations interchangeable.

The cost of that choice is scope. Anything a personal agent does that cannot be reduced to one verifiable token is outside what this benchmark can see, which covers most of the work you would actually delegate: writing the email, deciding whether the refund offer is good enough, judging whether to escalate. The benchmark is rigorous precisely because it declined to look at those.

What repeated attempts buy, and what that implies about cost

One finding in the launch evaluation deserves more attention than it gets. OpenAI generated 64 answers per question from Deep Research and picked one by majority vote, confidence-weighted vote or highest confidence. All three improved performance by 15% to 25% over a single attempt, and picking the most confident answer did best.

That is a large gain, and it is bought entirely with compute. Nothing about the model changed. It simply got more chances. The same paper reports calibration error, the distance between a model's stated confidence and its actual hit rate: 91% for Deep Research against 69% for GPT-4o without browsing. The system that found the most answers was also the most overconfident about the ones it got wrong.

For a benchmark leaderboard this is an interesting result about sampling. For anyone choosing a product it is a warning: a headline score can reflect a configuration that runs the same question many times over, and we have not established whether a consumer product offers that configuration or at what price. A number near the top of a table describes what was possible in an evaluation, not what happens when you ask your assistant something on a Tuesday.

What BrowseComp cannot tell you about a personal agent

Reading, not doing.

Every BrowseComp answer is a fact retrieved. Nothing is booked, cancelled, disputed, paid for or sent, and no wrong answer costs you anything except a wrong answer. The entire class of failures that makes a personal agent frightening, which is acting confidently on your behalf and getting it wrong, is outside the scope of the test.

BrowseComp questions do not measure a continuing user relationship; other benchmarks may include seeded history. There is no yesterday. A system that remembers you refuse red-eye flights, that has your account numbers, that learned last month which of your three email addresses is the real one, scores exactly the same as a system that knows nothing about you. Memory is worth zero points.

There is a third gap that is specific to this benchmark. BrowseComp rewards persistence, and persistence is not free. An agent that searches thirty times to find one fact is expensive and slow, and we have not established how long any consumer product lets an agent keep searching before it stops. The benchmark rewards a behaviour that is correct in a lab and may be impractical in a product, and its score carries no cost or time column to tell you which.

So what should you measure instead

If persistence and retrieval are not the questions, the questions are these. Where does the model run, and what can it reach on your machine. How does a tool get in front of it, and who decided which ones. Where does your memory live, and can you correct it. What do you get to approve, and how do you reach the thing at all. And what could nobody establish from the outside.

Those are the six axes on our scorecard, built by taking personal agents apart at the runtime level. The completion rates it carries came from recorded task runs in adapted product runtimes with a substituted exam model, each printed with an interval and the number of tasks behind it, and one row carries no number at all because the figure we were given does not reconcile with the records we can check.

FAQ

What is BrowseComp? A benchmark of 1,266 questions, each asking for a fact that is hard to find on the open web and simple to verify once found. It was built to measure whether a browsing agent keeps searching when the answer does not turn up quickly.

How hard is BrowseComp for humans? Very. In the verification campaign, trainers solved 367 of 1,255 problems, which is 29.2%, and gave up on 888 after about two hours each. Where they did answer, they matched the reference answer 86.4% of the time.

How is BrowseComp scored? Each question has a short reference answer. OpenAI's official evaluator asks a grader model whether the response's final answer matches it, and the headline figure is accuracy over the questions. The paper also reports calibration error from a confidence score each model gives.

What is the best BrowseComp score? It depends whose run you trust. On Kaggle's board, re-read on 16 October 2026, the top model was Gemini 3.1 Pro Preview at 31.2%, run in March 2026. At launch, OpenAI's Deep Research scored 51.5%. Moonshot AI's Kimi K3 model card reports 91.2% on its own setup. Vendor figures like that are self-reported and not comparable with Kaggle's.

Is there an official BrowseComp leaderboard? There is a Kaggle-maintained board listed under OpenAI's name on Kaggle, with 42 models and a last update in March 2026. Unlike GAIA, we found no way for outside teams to submit to it, and OpenAI's own launch figures are not on it. Rankings elsewhere are compilations, so ask where each row came from.

How is BrowseComp different from GAIA? GAIA has a human figure of 92%, and its self-submitted leaderboard is above it. BrowseComp's trainers solved 29.2%, Kaggle's best run is 31.2%, and vendor self-reports are above 90%, so how solved it is depends on whose harness you count. GAIA asks whether a system can finish a multi-step lookup. BrowseComp asks whether it will keep looking.

Does BrowseComp measure whether an agent can act on my behalf? No. Every answer is a fact retrieved. Nothing is booked, cancelled, sent or paid for, so the failure mode that matters most in a personal agent, acting wrongly with confidence, is not tested at all.

Why a BrowseComp score is not a personal agent score

BrowseComp is a good benchmark, built carefully, measuring something real that other benchmarks miss. A 51.5% at launch against a trainer solve rate of 29.2% was a genuine result, and the paper's footnote that Deep Research was trained on data meant to make it good at BrowseComp tasks is the kind of disclosure worth respecting.

It still tells you nothing about whether a product should be trusted with your accounts, because it never asks a system to remember you, act for you, or explain afterwards what it did. Those questions have their own page: how we score, the four teardowns, and what the other benchmarks measure.

The other benchmark pages

• GAIAEvery question arrives cold. Nothing carries over from the last one, so a system that remembers you scores exactly the same as one that doesn't.
• WebArenaThe sites aren't yours and hold nothing about you. No saved cards, no order history, no account that remembers.
• OSWorldSkill, not memory. It asks whether an agent can operate the machine, never whether it knows why you'd want it to.
• Tau-benchThe user is simulated and the domains are narrow, but this is the closest thing on the list to what a personal agent actually does.
• AgentBenchNone of the environments is your life. No calendar, no inbox, no bill to dispute.
• Terminal-BenchAlmost nothing a personal agent does happens at a terminal.
• SWE-benchA closed problem with a written specification. An errand has neither, and it cannot be rolled back.
• AssistantBenchStill one-shot. No standing preferences, no routine that fires next Monday.