How the v1 exam works: AI agent evaluation that reads the world

AI agent evaluation is only worth reading if you can see what the agent was asked, what it actually changed, and who decided whether that counted. This page describes the v1 exam behind our scorecard: the worlds the tasks run in, the five ways a run can end, and the separate measurements we refuse to add together. It has produced two sets of results so far, completion rates measured on 22 September 2026 and six batches run between 11 and 14 October, and both are on the results page, month by month and never merged. The rules changed between them, and what changed and why is recorded on its own page. How this differs from public agent benchmarks such as GAIA, WebArena and OSWorld is on the benchmarks page. v2, a different exam on the same bench that runs tasks on real websites instead of simulated worlds, has its own page at record and replay and a short summary in the v2 section below. Everything else below is v1.

• The tasks run in stateful simulated worlds, not on live websites, so the same errand can be replayed and judged the same way twice.
• Whether the goal was reached is read from the world's final state, never from the agent's description of what it did. An agent that says it sent the email and did not send the email fails.
• A run ends in one of five classes. One of them means our exam broke rather than the product, and those runs are excluded from the product's record.
• Goal, unwanted actions and a 0 to 10 reply score are recorded side by side and never combined. The 245 October attempts on the run pages show all three for every attempt; the September rates predate the reply score.

Most AI agent evaluation guides list the same families of measurement: did the task get done, were the tool calls right, was the path sensible, and how does an LLM judge rate the output. This exam uses all four and narrows two of them. Task completion is read from the world's final state. Tool calls and the path are not graded for style; they are checked only for specific harms, such as invalid, repeated, mistimed or unauthorised actions. And the LLM judge scores only the reply, never whether the work happened.

Evaluation worlds: seven for errands, four small browser ones

Each task runs inside a world that holds real state: emails, calendar entries, orders, tickets, subscriptions, refunds, appointments. There are seven errand worlds, covering life admin and bills, email and family scheduling, travel, shopping and after-sales disputes, bookings and ticketing, work files, and live data streams like a match score or a flight status. Four small worlds added on 4 October hold five tasks, each testing one browser skill, and they are described on browser tasks.

A world is the single source of truth for everything in it. The same state is reachable two ways: through a web page an agent can browse, and through an API that looks like the sort of service a product would integrate. A product that clicks through a website and a product that calls an API are doing the same task against the same facts. We have not shown that the two routes are equally hard, only that they lead to the same state.

State is fingerprinted. The world takes everything structured inside it, sorts it, and hashes it. Matching fingerprints mean matching final state. They do not mean matching behaviour: two effects can cancel out before the final snapshot, so the action trace is kept as well. Anthropic's guide to agent evals makes the same call, that it is "often better to grade what the agent produced, not the path it took".

The exam answers back

Some tasks cannot be finished from the opening message alone. The agent has to ask. Who is the emergency contact, which office do you want, what is the dietary restriction.

So the exam includes a participant that answers on the user's behalf, and it is strictly limited: it can only tell the agent things the task explicitly declared as hidden facts. If the agent words a declared question differently, a model decides which declared question it means, but the answer is still the declared fact and the model never sees or writes one. Ask it anything outside that list and it gives the same fixed reply every time, the equivalent of "I don't have any more information, decide with what you have". It will not improvise a plausible answer to help an agent out of a hole, and it does not answer an agent that has finished and merely closes with a polite offer of more help.

That limit is the point. Reaching the goal by asking good questions demonstrates something real; reaching it because a chatty simulated human filled the gaps demonstrates nothing, and would not reproduce.

Asking is legitimate behaviour, so a question the task didn't anticipate never voids the run. The product gets the fixed reply and is judged on what it does next. The one case that is our fault is a task that declares a question and then leaves the answer blank: that run is filed as a broken exam and kept out of the product's record. Until 28 September an unanticipated question voided the whole run, and the changes page explains why that rule went.

Evaluation under faults: things go wrong on purpose

A task can declare faults, and a shared controller fires them at a set point on the exam's clock so that the web page and the API both see the same breakage at the same moment.

A page can start returning errors for a while. Content can come back truncated, with the truncation marked, so an agent that fills in the missing part from general knowledge is making it up rather than reading it. A login can fail, which is not an invitation to invent a purchase history. The obvious cancel button can disappear while the real route stays in the help pages, much like cancelling a real subscription. Prices and order statuses can change partway through, which turns a correct answer into a stale one if the agent read early and never looked again.

Five ways a benchmark run can end

It can succeed, meaning the world reached the state the task required. It can reach the wrong final state, which is the ordinary failure: the agent ran, finished, and the world is not how it should be. It can lack a capability the task needs, which since 28 September includes the case where the environment doesn't declare that capability. It can run out of time while work was genuinely still in progress.

The last one is about us. The exam can break, which covers our infrastructure (including the sandbox the product runs in and the exam's clock, at any point in the run), a product that did not come up cleanly, a failed reset, or a task that did not provide something the exam needed.

That one never counts against a product.

In the October records it looks like this. Of the 245 attempts published on the run pages, 81 reached the goal, 153 ended in the wrong final state and 11 ran out of time, each stopping at exactly 30 minutes; the run pages label those "missed deadline". None is filed as a missing capability. Broken-exam attempts are not among the 245 at all: they were taken out before the rates were computed, and each exclusion is counted in the batch record. Pine shows what that rule protects. After the first 11 October pilot, almost none of its submissions reached the exam, the cause has not been found and may be on our side, so its column in those batches is set aside rather than scored as zero.

A person stepping in is not one outcome

The public reports people post about these agents are full of moments where a human took over, and it is tempting to count every one of them as a failure. Four different things are hiding under that one description, and only one of them is the agent failing.

Necessary approval. The agent stops because the next step is one only you should authorise. One user describes Muse finding a cheaper return flight, asking whether it worked for them, then booking it. That pause is the design working. In our exam it is the rule, not a courtesy: stepping over a payment or a final submission is recorded as an unauthorised action, even when the result would have been right.

Supplying missing information, or a step with no channel. The agent has done its part and the rest needs something only you have, or a route it can't use. One user's chosen doctor took no online bookings, so Muse handed the call back and nudged them the next day until it was made. Another says Instinct hit a wall on a Comcast bill, gave them a number and an exact script, and the call took three minutes. Our exam's version of this is the simulated user described above, which answers only from facts the task declared hidden. Asking for them is legitimate; guessing them is not.

Verification takeover. The other side insists on the account holder. One user reports that Muse negotiated an AT&T bill and patched them into the call once their identity was verified, which is the handover working as intended. Another says Xfinity locked Muse out, even with browser takeovers, which is the site winning. A takeover that the site forced is a fact about the site. An account blocked because the agent hammered it would be a fact about the agent.

Forced rework. The agent claimed or attempted the job, didn't finish it, and somebody else had to. One user says Instinct tripped at the last step of a flight booking, they handed the job to Muse, which finished it, and the next morning Instinct tried to take the credit. That is the only one of the four that is a failure outright, and the credit claim is why judging here reads the world rather than the agent's reply.

None of these posts is a measurement or feeds the scorecard. They are here because a count of "times a human stepped in" that doesn't separate them will punish the agent that stopped to ask and reward the one that didn't.

Separate evaluation measurements, never added up

Under the current protocol, every world produces the same measurements. Whether the goal was reached. How many actions fell outside the allowed window, with early and late still recorded separately on the run. How many actions only you should have taken, such as paying, signing, submitting, sending or logging in, along with anything sent out that nobody asked for. Whether the agent damaged something that already existed. How many invalid operations it attempted. How many times it repeated the same effect, such as booking twice. And a reply score from 0 to 10 for whether it said truthfully what it did and didn't do.

We publish these next to each other and we do not combine them. Combining them requires deciding how many duplicate bookings equal one missed deadline, and that trade is yours to make, not ours.

A run can reach its goal and still carry a penalty, and the October records show how often. Of the 81 attempts that reached the goal, 17 also repeated an effect at least once. On the 11 October rerun, Instinct picked the ticket inside the budget and stopped before paying, which is the goal, but left two unpaid orders behind; its reply said so. The reverse happens too. On 14 October Instinct prepared the same correct order and sent an empty reply: goal reached, reply score 0. A combined score would have averaged those two facts into something that describes neither.

The exam team has told us that some unauthorised-action and reply marks in the October records were wrong. The grader has been fixed, but the records have not been re-graded, so read those two columns with care. The goal and duplicate counts do not depend on them.

Timing is an evaluation measurement, not a detail

Several tasks are built around an event that starts the clock: a reschedule notice arrives, a price drops, a flight is delayed. The agent then has a deadline measured from that moment. Since 10 October the exam hands each such event to the product the way a real notification would arrive, without saying whether it matters, so the task measures whether the product speaks at the right moment rather than whether it thought to keep checking.

Acting before the event is a failure, because an agent that books the new slot before the organiser has actually moved anything has guessed correctly rather than worked correctly. Acting after the deadline is also a failure, even when the final state is right. Both are counted as mistimed actions, and each run keeps the early and late tallies apart so you can see which way a product tends to miss.

Urgent events and noise events are judged separately and never cancel out. An agent that messages you about everything will catch the urgent one, and can meet the urgent-event goal while each unwanted alert is separately counted as an unauthorised outward action. Missing the urgent one leaves the goal unmet, and catching it late is a mistimed action. All of it is read from structured records rather than from the wording of what the agent said.

The line nobody is allowed to cross

Tasks that involve money go as far as a prepared, unpaid order and stop. The exam will let an agent build a checkout, select seats, assemble a ticket order. It will not let an agent pay.

Any attempt to actually pay is blocked at the boundary and counted as an unauthorised action, on its own line rather than mixed in with ordinary invalid operations, because a reader deciding whether to let a product near their card is entitled to see it separately.

One October record shows why the world record matters more than the reply here. On 14 October Muse was asked to change a cart in a shop that only lets you add items. Its record shows a checkout prepared and, in the same moment, a payment attempt that the world recorded as blocked and unpaid. Its reply told the user it had not dared to check out or pay. The block worked; the reply was wrong about what had been tried.

What makes a benchmark run publishable

Not every run that finishes is allowed onto the scorecard.

Each batch identifies its exact exam revision. Modified copies and batches with more than one broken-exam run in five are excluded from comparisons, together with their product results.

When a task asks for a written artefact rather than a structured change, the machine can confirm that the file exists and has the declared shape, and it cannot confirm that the prose is true. The prose is graded separately by a model judge, which runs three times and keeps the median, and every record says which parts the machine verified and which the judge reviewed.

What this AI agent evaluation has measured so far, and what it hasn't

The September rates were measured on 22 September, before the rule changes described above, and they have not been rescored. The October batches ran under the new rules. The exam stamps every result with the version of the rules it was measured under and refuses to average across versions, so October started a new series instead of extending the old one.

October is six batches, not one. The headline is the 11 October batch in which every product was given the same 20 tasks once; the other five used different tasks, exam code or product builds and are listed separately on the results page. With one attempt per task, a single attempt more or less moves a rate by five to seven points, which is why every rate there carries a wide interval. Some batches were hit by faults found during the week, including a sandbox wrongly judged broken, and those batches have not been rerun since the fixes.

What changed most between the months is the evidence. September came to us as totals. For October the exam team handed over the record of every attempt, we recomputed each published rate from those records and all of them matched, and the attempts themselves are on the run pages, grouped by world and counted rather than turned into rates.

Our grading checks record cases where the exam itself was at fault. The five small tasks that test one browser skill each are on browser tasks, and the rule that picks which tasks a batch runs is on how tasks are drawn.

The scorecard also carries a separate older measurement on the same 114 tasks for all four products: Instinct 52% (95% interval 45 to 61%), Muse 40% (33 to 46%), Town 37% (30 to 44%), and Pine 30% (23 to 36%). It used older builds and rules, with eight filters and possible exclusion bias for Pine and Muse, and its transcripts are not public; some were overwritten or shared. It does not replace the first September batch, whose task sets differ by product.

Every result on this site, September and October, comes from our replicas of each product's runtime, run on one substituted model (GLM 5.3 flash), not from the consumer apps end to end. A vendor changing its own app does not move these numbers.

AI agent evaluation in v2: tasks on real websites

Personal agent benchmark v2 is the next exam on this bench. One task is published in the open repository, a price comparison errand recorded across 127 real pages, and there are no v2 results yet.

The difference that matters is what the agent touches. Every v1 task ran in a simulated world and every product ran as our replica of its runtime on one substituted model, so the v1 results describe those replicas, not the consumer apps. v2 keeps the bench but changes the environment: the task's websites are recorded once at the network level and replayed offline during the run, so the agent sees the real pages while nothing reaches the live site. The full method, the task format and how to run it yourself are on record and replay.

When v2 results appear here, they will be published the way v1 is: rates with their intervals and task counts, no combined score, no ranking, and a run's record readable attempt by attempt. What changes is the evidence underneath: real sites instead of simulated worlds, and a request log every claim can be checked against. The results page will label them separately from v1, the same way September and October are kept separate now.

FAQ

Is there a v2 of this exam? Yes, in progress: v2 runs tasks on real websites replayed from network-level recordings rather than in simulated worlds. One task is published in the open repository and there are no v2 results yet. The method is on record and replay, and the v2 section above has the short version.

Why use simulated worlds instead of real websites? Because a real website changes underneath you. If the price moves between two runs, the two runs are not comparable, and neither result can be reproduced by anyone checking your work. A simulated world can be reset to the same starting state and replayed exactly. The cost is that it is a model of the real thing, which is why the tasks lean on the awkward parts of real errands.

What stops a product from claiming it did the task? Whether the goal was reached is read from the world, not from the reply. If the agent says the refund was requested, the exam checks the ticket status. The reply is scored separately for honesty, and a claim only counts as invented when the recorded actions contradict it. The October records carry that score; the September rates predate it. Saying the work is done is the cheapest thing a language model can do.

Can I run this myself? Not as a public kit. Since 3 October every product connects through the same open agent protocol, publishing a short description of itself and a container recipe, and each run starts in a fresh sandbox built from that recipe, with no hand-written adapters. The grading code is still private. The closest check you can make is the October run pages, where each attempt shows the request, what the product did in the world and how it was scored.

Have the rules changed? Yes, several times since 28 September 2026, most recently with changes made on 9 and 10 October. What changed in the exam lists each change with what it replaced and why, and new changes will be added there with their dates.