How the exam works
A personal agent benchmark is only worth reading if you can see what the agent was asked, what it actually changed, and who decided whether that counted. Most agent leaderboards show you a number and a model name. This page describes the exam behind our scorecard instead: the worlds the tasks run in, the six ways a run can end, and the five measurements we refuse to add together.
- The tasks run in stateful simulated worlds, not on live websites, so the same errand can be replayed and judged the same way twice.
- Judging reads the world's final state, never the agent's own description of what it did. An agent that says it sent the email and did not send the email fails.
- A run ends in one of six classes. Two of them mean our exam broke rather than the product, and those are excluded from the product's record.
- Five measurements come out of every task, and we publish them side by side rather than combining them into a score.
Seven worlds that remember what you did
Each task runs inside a world that holds real state: emails, calendar entries, orders, tickets, subscriptions, refunds, appointments. There are seven of them, covering life admin and bills, email and family scheduling, travel, shopping and after-sales disputes, bookings and ticketing, work files, and live data streams like a match score or a flight status.
A world is the single source of truth for everything in it. The same state is reachable two ways: through a web page an agent can browse, and through an API that looks like the sort of service a product would integrate. Both read and write the same underlying record, which matters more than it sounds. A product that clicks through a website and a product that calls an API are doing the same task against the same facts, so neither gets an easier exam because of how it happens to be built.
State is fingerprinted. The world takes everything structured inside it, sorts it, and hashes it. Two runs that produce the same fingerprint changed the same things. That is what makes a run replayable rather than merely recorded.
The exam answers back
Some tasks cannot be finished from the opening message alone. The agent has to ask. Who is the emergency contact, which office do you want, what is the dietary restriction.
So the exam includes a participant that answers on the user's behalf. It is deterministic and it is strictly limited: it can only tell the agent things the task explicitly declared as hidden facts. Ask it anything outside that list and it says it does not know, every time, in the same words. It will not improvise a plausible answer to help an agent out of a hole.
That limit is the point. An agent that reaches the goal by asking good questions has demonstrated something real. An agent that reaches the goal because a chatty simulated human filled in the gaps has demonstrated nothing, and the next run would not reproduce it.
There is a matching rule for the other direction. If a product asks a question on a task that never declared a simulated user, that is our fault, not the product's. Asking is legitimate behaviour. The exam was short of a participant. Those runs are filed under exam gaps and kept out of the product's record entirely.
Things go wrong on purpose
A task can declare faults, and a shared controller fires them at a set logical time so that the web page and the API both see the same breakage at the same moment.
A page can start returning errors for a while. Content can come back truncated, with the truncation marked, so an agent that fills in the missing part from general knowledge is making it up rather than reading it. A login can fail, which is a statement about account boundaries and not an invitation to invent a purchase history. The obvious cancel button can disappear while the real route stays available in the help pages, which is a fairly accurate model of cancelling a subscription in real life.
Prices and order statuses can change partway through, which turns a correct answer into a stale one if the agent read early and never looked again.
Six ways a run can end
Every run is filed under exactly one outcome, and the wording is deliberately blunt about whose fault it was.
It can succeed, meaning the world reached the state the task required. It can reach the wrong final state, which is the ordinary failure: the agent ran, finished, and the world is not how it should be. It can lack a capability the task needs. It can run out of time while work was genuinely still in progress.
The last two are about us. The exam can break, which covers our infrastructure, a product that did not come up cleanly, a failed reset, or a task that did not provide something the exam needed. Or the environment can be missing a capability the task requires, such as the ability to advance time.
Those last two never count against a product. A bench that quietly logs its own infrastructure failures as product failures is measuring its own reliability and calling it someone else's score.
Five measurements, never added up
Each task produces five numbers that every world defines, and some worlds add their own.
The universal five are: whether the goal was reached, whether the agent acted before it was allowed to, whether it acted after the deadline, whether it damaged something the task was watching, and how many invalid operations it attempted. Worlds add their own on top. Shopping tracks whether the agent crossed the payment line. Live data worlds track whether it interrupted you over something that did not warrant it.
We publish these next to each other and we do not combine them. Combining them requires deciding how many duplicate bookings equal one missed deadline, and that trade is yours to make, not ours. Somebody who wants an agent that never spends money without asking and somebody who wants an agent that finishes the errand unsupervised are shopping for different products, and a single blended score hides exactly the difference they care about.
A run can reach its goal and still carry a penalty. Read the columns separately, because that combination is often the most informative result on the page.
Timing is a measurement, not a detail
Several tasks are built around an event that starts the clock: a reschedule notice arrives, a price drops, a flight is delayed. The agent then has a deadline measured from that moment.
Acting before the event is its own failure, recorded separately, because an agent that books the new slot before the organiser has actually moved anything has guessed correctly rather than worked correctly. Acting after the deadline is also its own failure, even when the final state is right.
Urgent events and noise events are judged separately and never cancel out. An agent that messages you about everything will catch the urgent one, and it does not get credit for that, because it also interrupted you six times over nothing. Both are read from structured records rather than from the wording of what the agent said.
The line nobody is allowed to cross
Tasks that involve money go as far as a prepared, unpaid order and stop. The exam will let an agent build a checkout, select seats, assemble a ticket order. It will not let an agent pay.
Any attempt to actually pay is blocked at the boundary and counted on its own. It is not mixed in with ordinary invalid operations, because it is a different kind of mistake and a reader deciding whether to let a product near their card is entitled to see it on its own line.
The same logic covers actions that only a person can legitimately take: signing something, approving a draft, reading a code out of a message. An agent can prepare all of it. Stepping over the final submission is recorded as acting too early rather than as success.
What makes a run publishable
Not every run that finishes is allowed onto the scorecard.
A batch records the exact commit it ran from and whether the checkout was clean. A batch run from a modified working copy is marked non canonical, and it stays readable as evidence while being excluded from every comparison. There is also a quality gate on the batch as a whole: if too large a share of its runs were broken exams rather than real attempts, the whole batch is refused, and refusal is inherited by everything underneath it. A product result cannot quietly declare itself scored inside a batch that failed.
There is one more distinction worth knowing, because it is the sort of thing benches usually hide. When a task asks for a written artefact rather than a structured change, the machine can confirm that the file exists and has the declared shape, and it cannot confirm that the prose is true. Those runs are labelled as structurally verified with the wording still unreviewed, and they stay out of the automated success figures until a person has read them.
Why the scorecard still says not yet scored
Everything above is built and runs. What has not happened yet is the part that matters: handing the same errands to all four products, under the same conditions, enough times for the numbers to mean something.
We would rather show you the exam first. A benchmark that publishes its scoring rules after its scores has already chosen its conclusion. When the runs exist, every one of them gets a page showing the conversation, what changed in the world, which pages were opened, and the five measurements, including the runs that went nowhere.
FAQ
Why use simulated worlds instead of real websites? Because a real website changes underneath you. If the price moves between two runs, the two runs are not comparable, and neither result can be reproduced by anyone checking your work. A simulated world can be reset to the same starting state and replayed exactly. The cost is that it is a model of the real thing rather than the real thing, which is why the tasks are built around the awkward parts of real errands: outages, truncated pages, hidden cancellation routes.
Does the agent know it is being tested? It is given a task the way a user would give it, and nothing in the opening message announces an exam. It is talking to a normal looking web page or API.
What stops a product from claiming it did the task? Judging never reads the agent's reply. It reads the world. If the agent says the refund was requested, the exam checks the ticket status. This is the single most important rule in the whole design, because saying the work is done is the cheapest thing a language model can do.
Why publish failures at all? A bench that only reports wins is marketing with a table in it. The failures are also the most useful part for a reader, because they tell you the shape of the thing you will be cleaning up after.
Can I run this myself? The exam runs products through a narrow adapter layer, so connecting a product means implementing a small interface rather than modifying the bench. That is how we intend to keep this checkable rather than asking you to trust our numbers.