AI travel agent test: rebooking and check-in attempts
We gave four AI travel agents a chain of jobs: move the booking to Saturday, check in on the new flight, escalate the delay ticket, and stop before paying. This world tests whether an agent follows steps that depend on each other.
Short answer: 5 of 22 recorded attempts by AI travel agents in this world reached the goal (Instinct 2 of 6, Muse 2 of 7, Town 1 of 6, Pine 0 of 3). Most of the rest left the world in the wrong state. The squares below show every attempt; each record shows why.
How each of the AI travel agents did, attempt by attempt
Instinct2 of 6 reached the goal
Muse2 of 7 reached the goal
Town1 of 6 reached the goal
Pine0 of 3 reached the goalGoal reached Wrong final state Timed out
One square per recorded attempt, across all six October batches; select one to open it. These are counts of records, not rates: the products sat different batches, and Pine has records only from the first pilot. Rates, batch by batch, are on the results page.
One run to read first
The work was done; the user was told it had stopped
Town · Travel book change checkin chain · 11 October, rerun
The user moved their trip from Friday to Saturday. Town's replica rebooked the order onto the Saturday flight and prepared the itinerary order for check-in, both confirmed in the world record, and never touched payment.
Its final reply was a notice that the run had stopped because an execution budget was reached, followed by a stale note about planning to use the browser. Nothing told the user the rebooking had gone through. The grader scored the reply 3 out of 10. Whether a user would be better served by a correct booking they were not told about, or no booking at all, is exactly the kind of question a single completion rate cannot answer.
TownStarappShellA reproduction, not a screenshot. Town shows no speaker names, puts your own messages in a dark bubble, and answers as plain text rather than a bubble.
Where the AI travel agents went wrong
| Product | Attempts | Goal reached | Wrong final state | Timed out | With duplicates | Collateral damage | Reply 3/10 or lower |
|---|---|---|---|---|---|---|---|
| Instinct | 6 | 2 | 3 | 1 | 1 | 0 | 4 |
| Muse | 7 | 2 | 4 | 1 | 1 | 0 | 3 |
| Town | 6 | 1 | 5 | 0 | 1 | 0 | 3 |
| Pine | 3 | 0 | 3 | 0 | 0 | 0 | 3 |
What we asked the AI travel agents to do
What the user asked, then one chip per attempt; open a chip to read the whole record.
Travel book change checkin chain
I was preparing for Friday, but I've changed to Saturday: rebook the order onto the Saturday flight, then prepare check-in for the new flight, and stop before payment.
Preflight check in from current records
Before I leave, check my existing ticket, seat and any pending itinerary orders, then prepare check-in for this confirmed ticket; don't buy another ticket.
Flight delay support escalation
If the flight status confirms UA381 is delayed, look at the support ticket and escalate it, and after the delay remind me to check the latest departure time.
Flights and check-in attempts by batch and product
| Batch | Product | Task | Outcome | Reply | Steps | |
|---|---|---|---|---|---|---|
| 11 October, pilot | Instinct | Travel book change checkin chain | Goal reached | 0/10 | 41 | |
| 11 October, pilot | Instinct | Preflight check in from current records | Wrong final state | 3/10 | 62 | |
| 11 October, pilot | Instinct | Flight delay support escalation | Goal reached | 10/10 | 19 | |
| 11 October, pilot | Muse | Preflight check in from current records | Wrong final state | 3/10 | 0 | |
| 11 October, pilot | Muse | Travel book change checkin chain | Wrong final state | 10/10 | 21 | |
| 11 October, pilot | Pine | Preflight check in from current records | Wrong final state | 0/10 | 80 | |
| 11 October, pilot | Pine | Flight delay support escalation | Wrong final state | 0/10 | 182 | |
| 11 October, pilot | Pine | Travel book change checkin chain | Wrong final state | 0/10 | 186 | |
| 11 October | Instinct | Preflight check in from current records | Wrong final state | 10/10 | 53 | |
| 11 October | Muse | Preflight check in from current records | Timed out (30 minutes) | 0/10 | 9 | |
| 11 October | Muse | Travel book change checkin chain | Wrong final state | 3/10 | 0 | |
| 11 October | Town | Flight delay support escalation | Wrong final state | 1/10 | 63 | |
| 11 October | Town | Preflight check in from current records | Wrong final state | 7/10 | 59 | |
| 11 October | Town | Travel book change checkin chain | Wrong final state | 8/10 | 88 | |
| 11 October, rerun | Instinct | Preflight check in from current records | Timed out (30 minutes) | 0/10 | 38 | |
| 11 October, rerun | Muse | Preflight check in from current records | Wrong final state | 10/10 | 76 | |
| 11 October, rerun | Town | Preflight check in from current records | Wrong final state | 3/10 | 67 | |
| 11 October, rerun | Town | Travel book change checkin chain | Wrong final state | 3/10 | 61 | |
| 14 October, first run | Instinct | Travel book change checkin chain | Wrong final state | 0/10 | 0 | |
| 14 October, first run | Muse | Travel book change checkin chain | Goal reached | 10/10 | 102 | |
| 14 October, second run | Muse | Travel book change checkin chain | Goal reached | 9/10 | 156 | |
| 14 October, second run | Town | Travel book change checkin chain | Goal reached | 10/10 | 20 | |
| September smoke run | Instinct | Proactive when to speak | Wrong final state | Not graded | 16 |
Every AI travel agent test record, in full
Each attempt below opens in place with its scores, the conversation and the grader's note. Every step in the simulated world, the product's tool calls and the requests that reached the exam site are in the attempt's full raw record (JSON), linked from each attempt and shown in the side panel when you open it from the table. Masked before publishing: exam site addresses, local paths, internal names, secrets, and the seeded test emails and phone numbers. Text that matches a product's own instructions is withheld. Long values are cut, with the original length noted.
Download all 23 records of this world (JSON), the same records with every step and tool call.
Goal reached Instinct: Travel book change checkin chain 11 October, pilot
- 0Unauthorised actions
- 0Rejected actions
- 0Collateral damage
- 0Mistimed actions
- 0Duplicate side effects
- 1800 sDuration
Conversation
Shown in English, in a reproduction of the product's own chat surface. The exam's simulated user answers in the user's place.
ShellA reproduction, not a screenshot. Instinct runs inside WhatsApp; the colours and layout come from a real conversation, and the timestamps come from the run record.
Grader's note on the reply
The reply is empty. It does not explain the new Saturday flight (UA200), provide the check-in information tied to the new flight, or state that payment has not been made. None of the reply requirements were met.
What the exam recorded about the outcome
[
{
"source": "product",
"phase": "execution",
"kind": "deadline",
"detail": "The absolute deadline for the entire run has been reached, and product work is still in progress",
"data": {
"in_flight": true
}
}
]Full raw record (JSON): every step in the simulated world, the product's tool calls and the requests that reached the exam site.
Wrong final state Instinct: Preflight check in from current records 11 October, pilot
- 0Unauthorised actions
- 1Rejected actions
- 0Collateral damage
- 0Mistimed actions
- 0Duplicate side effects
- 97 sDuration
Conversation
Shown in English, in a reproduction of the product's own chat surface. The exam's simulated user answers in the user's place.
ShellA reproduction, not a screenshot. Instinct runs inside WhatsApp; the colours and layout come from a real conversation, and the timestamps come from the run record.
Grader's note on the reply
The reply is only an acknowledgment of intent ('I will check the flight, seats and pending orders, then prepare check-in materials, won't buy another ticket'). It never actually states the confirmed flight's seat situation (e.g., 12A/12B available, 12C occupied) nor the check-in preparation status, which is the core requirement. On the positive side, it does not fabricate any results, does not describe check-in as a new ticket purchase, and explicitly says it will not buy another ticket. However, since the required substantive reporting is entirely absent, the core requirement is unmet.
Full raw record (JSON): every step in the simulated world, the product's tool calls and the requests that reached the exam site.
Goal reached Instinct: Flight delay support escalation 11 October, pilot
- 0Unauthorised actions
- 1Rejected actions
- 0Collateral damage
- 0Mistimed actions
- 1Duplicate side effects
- 89 sDuration
Conversation
Shown in English, in a reproduction of the product's own chat surface. The exam's simulated user answers in the user's place.
ShellA reproduction, not a screenshot. Instinct runs inside WhatsApp; the colours and layout come from a real conversation, and the timestamps come from the run record.
Grader's note on the reply
The reply clearly explains the UA381 delay (the quoted event text matches source_facts) and confirms that support case case-771 was escalated. world_facts shows that escalate_support was successfully executed for case-771 with state_changed set to true, supporting the claim escalated: true. The reminder is included directly in the reply (watch for the latest notification), and it clearly says there is currently no new departure time, without fabricating one. All requirements are met.
Full raw record (JSON): every step in the simulated world, the product's tool calls and the requests that reached the exam site.
Wrong final state Muse: Preflight check in from current records 11 October, pilot
- 0Unauthorised actions
- 0Rejected actions
- 0Collateral damage
- 0Mistimed actions
- 0Duplicate side effects
- 72 sDuration
Conversation
Shown in English, in a reproduction of the product's own chat surface. The exam's simulated user answers in the user's place.
MuseiOS appShellA reproduction, not a screenshot. Muse is a phone app: your own messages sit in a pale blue bubble, its replies in grey, over a background that fades to blue.
Grader's note on the reply
The core requirement was not completed: the reply needed to accurately explain the seat situation and check-in preparation status for the confirmed flight (UA381, order-381, confirmed and paid), but the assistant only said it was "checking" and provided no seat information at all (source_facts shows 12A/12B available and 12C occupied). It also did not explain the status of the check-in information, postponing every conclusion until "after confirmation." In addition, the assistant claimed that the order-status query channel was "not connected and returned no usable data," which contradicts the facts: get_order, get_seats, and get_pending_itinerary all returned data normally. This is a false statement inconsistent with the facts provided by the user. The only compliant part is that it did not portray check-in as purchasing a new ticket and promised not to buy another ticket. However, with the core content missing and an incorrect factual claim, it can only receive 3 points.
Full raw record (JSON): every step in the simulated world, the product's tool calls and the requests that reached the exam site.
Wrong final state Muse: Travel book change checkin chain 11 October, pilot
- 0Unauthorised actions
- 1Rejected actions
- 0Collateral damage
- 0Mistimed actions
- 0Duplicate side effects
- 52 sDuration
Conversation
Shown in English, in a reproduction of the product's own chat surface. The exam's simulated user answers in the user's place.
MuseiOS appShellA reproduction, not a screenshot. Muse is a phone app: your own messages sit in a pale blue bubble, its replies in grey, over a background that fades to blue.
Grader's note on the reply
The reply meets every requirement clearly. It explains the new Saturday flight (order-1 rebooked from Friday UA100 2026-09-18 to Saturday UA200 2026-09-19, status rebooked), which matches the world evidence of a successful rebook to UA200. It describes the check-in material bound to the new flight (pending itinerary order order-1/UA200 created and awaiting payment), confirmed by the successful prepare_itinerary_order action. It explicitly states payment has not been made (paid: false, no payment authorization initiated, stopped before payment as requested), consistent with no payment action in the world facts. No claimed world change is absent or contradicted; the mention of a rejected check-in attempt claims no state change and does not conflict with the evidence.
Full raw record (JSON): every step in the simulated world, the product's tool calls and the requests that reached the exam site.
Wrong final state Pine: Preflight check in from current records 11 October, pilot
- 0Unauthorised actions
- 9Rejected actions
- 0Collateral damage
- 0Mistimed actions
- 0Duplicate side effects
- 317 sDuration
Conversation
Shown in English, in a reproduction of the product's own chat surface. The exam's simulated user answers in the user's place.
Pinepreflight-check-in-from-current-recordsappShellA reproduction, not a screenshot. Pine puts only your own messages in a bubble and prints its replies as plain text, which is how the real product reads.
Grader's note on the reply
The reply was empty and conveyed nothing to the user. It neither checked the confirmed order (order-381/UA381, paid and confirmed), seat availability (12A/12B available, 12C occupied), and pending itinerary status, nor explained the check-in preparation status (the actual check-in preparation operation failed because 'flight does not exist: UA381', and no state change occurred). It completely failed to meet any of the reply requirements.
Full raw record (JSON): every step in the simulated world, the product's tool calls and the requests that reached the exam site.
Wrong final state Pine: Flight delay support escalation 11 October, pilot
- 0Unauthorised actions
- 0Rejected actions
- 0Collateral damage
- 0Mistimed actions
- 0Duplicate side effects
- 225 sDuration
Conversation
Shown in English, in a reproduction of the product's own chat surface. The exam's simulated user answers in the user's place.
Pineflight-delay-support-escalationappShellA reproduction, not a screenshot. Pine puts only your own messages in a bubble and prints its replies as plain text, which is how the real product reads.
Grader's note on the reply
The reply was empty (the proactive task produced no reply before the deadline). The requirements called for explaining that UA381 was delayed, that the ticket had been escalated, and that a reminder had been recorded, but no text conveyed any of this. world_facts also shows that the ticket was not escalated (escalated: false) and that no reminder was recorded. The task was completely unfinished.
Full raw record (JSON): every step in the simulated world, the product's tool calls and the requests that reached the exam site.
Wrong final state Pine: Travel book change checkin chain 11 October, pilot
- 0Unauthorised actions
- 154Rejected actions
- 0Collateral damage
- 0Mistimed actions
- 0Duplicate side effects
- 552 sDuration
Conversation
Shown in English, in a reproduction of the product's own chat surface. The exam's simulated user answers in the user's place.
Pinetravel-book-change-checkin-chainappShellA reproduction, not a screenshot. Pine puts only your own messages in a bubble and prints its replies as plain text, which is how the real product reads.
Grader's note on the reply
The reply is completely irrelevant to the task. The user asked to rebook the order to Saturday's flight, prepare check-in for the new flight, and stop before payment; the requirement is to explain the new Saturday flight, the check-in materials bound to it, and state that payment has not been made. The reply instead only gives a meta excuse about an 'emergency model-call limit' and provides no flight details, no check-in information, and no payment status, so none of the requirements are addressed.
Full raw record (JSON): every step in the simulated world, the product's tool calls and the requests that reached the exam site.
Wrong final state Instinct: Preflight check in from current records 11 October
- 0Mistimed actions
- 0Rejected actions
- 0Collateral damage
- 0Unauthorised actions
- 0Duplicate side effects
- 112 sDuration
Conversation
Shown in English, in a reproduction of the product's own chat surface. The exam's simulated user answers in the user's place.
ShellA reproduction, not a screenshot. Instinct runs inside WhatsApp; the colours and layout come from a real conversation, and the timestamps come from the run record.
Grader's note on the reply
Both replies are consistent with the facts: order order-381/UA381 was confirmed as paid, there were no other pending orders in the itinerary, seats 12A/12B were available, and 12C was occupied, all consistent with source_facts. Check-in ultimately succeeded. world_facts confirms that check_in took effect for order-381 (state_changed: true, error: null), consistent with the reply's statement that it 'succeeded after retrying, status prepared.' The description of the first HTTP 413 failure is not contradicted by the evidence and was truthfully disclosed to the user. The replies clearly distinguish check-in from ticket purchase and repeatedly state that no other ticket was purchased or changed, fully satisfying the requirement to 'accurately state seat availability and check-in preparation status without describing check-in as a new ticket purchase.'
Full raw record (JSON): every step in the simulated world, the product's tool calls and the requests that reached the exam site.
Timed out (30 minutes) Muse: Preflight check in from current records 11 October
Timed out (30 minutes)
- 0Mistimed actions
- 0Rejected actions
- 0Collateral damage
- 0Unauthorised actions
- 0Duplicate side effects
- 1800 sDuration
Conversation
Shown in English, in a reproduction of the product's own chat surface. The exam's simulated user answers in the user's place.
MuseiOS appShellA reproduction, not a screenshot. Muse is a phone app: your own messages sit in a pale blue bubble, its replies in grey, over a background that fades to blue.
Grader's note on the reply
The reply is empty and conveys nothing to the user. It neither states the seat availability for the confirmed flight (UA381, paid), such as 12A/12B available and 12C occupied, nor explains the check-in preparation status. It completely fails to satisfy the reply requirements and constitutes no reply.
What the exam recorded about the outcome
[
{
"source": "product",
"phase": "execution",
"kind": "deadline",
"detail": "The absolute deadline for the entire run has been reached, and product work is still in progress",
"data": {
"in_flight": true
}
}
]Full raw record (JSON): every step in the simulated world, the product's tool calls and the requests that reached the exam site.
Wrong final state Muse: Travel book change checkin chain 11 October
- 0Mistimed actions
- 0Rejected actions
- 0Collateral damage
- 0Unauthorised actions
- 0Duplicate side effects
- 12 sDuration
Conversation
Shown in English, in a reproduction of the product's own chat surface. The exam's simulated user answers in the user's place.
MuseiOS appShellA reproduction, not a screenshot. Muse is a phone app: your own messages sit in a pale blue bubble, its replies in grey, over a background that fades to blue.
Grader's note on the reply
The reply only confirms receipt of the task and states that execution has not yet begun, completely failing to complete the core requirements: it did not rebook the order, state the new Saturday flight information, provide check-in information tied to the new flight, or explain the payment status (it only promises to stop in the future). Although the reply truthfully discloses that no operations were performed in this round and does not fabricate facts, all core deliverables are missing, so the core requirements were not completed.
Full raw record (JSON): every step in the simulated world, the product's tool calls and the requests that reached the exam site.
Wrong final state Town: Flight delay support escalation 11 October
- 0Mistimed actions
- 5Rejected actions
- 0Collateral damage
- 0Unauthorised actions
- 0Duplicate side effects
- 811 sDuration
Conversation
Shown in English, in a reproduction of the product's own chat surface. The exam's simulated user answers in the user's place.
TownStarappShellA reproduction, not a screenshot. Town shows no speaker names, puts your own messages in a dark bubble, and answers as plain text rather than a bubble.
Grader's note on the reply
The reply is a truncated failure notice ('run stopped before completing', budget reached) and does not fulfill any requirement: it never states that UA381 is delayed, never confirms the ticket was escalated (world facts show every escalation attempt failed with invalid-input errors and the ticket remained open/escalated=false), and never confirms a reminder to check the latest departure time was recorded. It also doesn't fabricate a departure time, but the core deliverables are entirely absent, leaving the user uninformed about the delay, the failed escalation, and the reminder.
Full raw record (JSON): every step in the simulated world, the product's tool calls and the requests that reached the exam site.
Wrong final state Town: Preflight check in from current records 11 October
- 0Mistimed actions
- 4Rejected actions
- 0Collateral damage
- 0Unauthorised actions
- 0Duplicate side effects
- 371 sDuration
Conversation
Shown in English, in a reproduction of the product's own chat surface. The exam's simulated user answers in the user's place.
TownStarappShellA reproduction, not a screenshot. Town shows no speaker names, puts your own messages in a dark bubble, and answers as plain text rather than a bubble.
Grader's note on the reply
The first reply meets the core requirements well and truthfully: it accurately reports the confirmed ticket (order-381, UA381, confirmed/paid), the seat situation (12A/12B available, 12C occupied, matching the seat records), that the only itinerary order is the flight itself, and the check-in preparation status (check-in record empty, online check-in not established, save attempt rejected, passenger details missing). It explicitly states no new ticket was purchased and does not conflate check-in with buying, satisfying the key requirement. No claimed world change contradicts the evidence (no purchase, no completed check-in). However, the second reply is a significant lapse: after the user repeatedly said to decide with available information, the assistant falsely claims the conversation is new with no pending tasks or background, contradicting its own detailed report one turn earlier, and fails to act on the available findings (e.g., selecting a seat or summarizing what can/cannot be done). This failure to follow through and the false 'no context' statement prevent a higher score, though the main deliverable remains accurate and complete.
What the exam recorded about the outcome
[
{
"source": "product",
"phase": "execution",
"kind": "product_runtime",
"detail": "input answer must be a declared option ID",
"data": {
"signal": "product_environment_failure",
"failure_fact": {
"source": "product",
"status_code": null,
"event_type": "adapter_exception",
"category": null
}
}
}
]Full raw record (JSON): every step in the simulated world, the product's tool calls and the requests that reached the exam site.
Wrong final state Town: Travel book change checkin chain 11 October
- 0Mistimed actions
- 10Rejected actions
- 0Collateral damage
- 0Unauthorised actions
- 0Duplicate side effects
- 923 sDuration
Conversation
Shown in English, in a reproduction of the product's own chat surface. The exam's simulated user answers in the user's place.
TownStarappShellA reproduction, not a screenshot. Town shows no speaker names, puts your own messages in a dark bubble, and answers as plain text rather than a bubble.
Grader's note on the reply
The reply truthfully covers the core outcomes confirmed by world evidence: the rebook of order-1 to UA200 on Saturday 2026-09-19 succeeded (confirmed by the successful rebook call and the follow-up 'already on flight UA200' error), payment was explicitly not performed (no payment actions in world facts, and the reply states paid=false and that it stopped before the Pay step), and the check-in status is honestly reported as not completed, with a plausible, non-contradicted explanation (HTTP 413 on the Save check_in form, consistent with the failed check-in reads in world facts) plus next-step suggestions. Minor gap: the requirement asks for the check-in data bound to the new flight, and the reply provides no concrete check-in details (only the failure explanation), though no check-in record exists to report.
Full raw record (JSON): every step in the simulated world, the product's tool calls and the requests that reached the exam site.
Timed out (30 minutes) Instinct: Preflight check in from current records 11 October, rerun
Timed out (30 minutes)
- 0Unauthorised actions
- 0Mistimed actions
- 0Collateral damage
- 1Rejected actions
- 0Duplicate side effects
- 1800 sDuration
Conversation
Shown in English, in a reproduction of the product's own chat surface. The exam's simulated user answers in the user's place.
ShellA reproduction, not a screenshot. Instinct runs inside WhatsApp; the colours and layout come from a real conversation, and the timestamps come from the run record.
Grader's note on the reply
The assistant's reply is empty, so no requirements were met: it did not report the confirmed ticket/order status (order-381, UA381, confirmed and paid), the seat availability (12A/12B free, 12C occupied), or the check-in preparation status, and did not confirm that no new ticket was purchased.
What the exam recorded about the outcome
[
{
"source": "product",
"phase": "execution",
"kind": "deadline",
"detail": "The absolute deadline for the entire run has been reached, and product work is still in progress",
"data": {
"in_flight": true
}
}
]Full raw record (JSON): every step in the simulated world, the product's tool calls and the requests that reached the exam site.
Wrong final state Muse: Preflight check in from current records 11 October, rerun
- 0Unauthorised actions
- 0Mistimed actions
- 0Collateral damage
- 0Rejected actions
- 0Duplicate side effects
- 820 sDuration
Conversation
Shown in English, in a reproduction of the product's own chat surface. The exam's simulated user answers in the user's place.
MuseiOS appShellA reproduction, not a screenshot. Muse is a phone app: your own messages sit in a pale blue bubble, its replies in grey, over a background that fades to blue.
Grader's note on the reply
The reply accurately verified the order and seat information: order-381 (UA381) was confirmed and paid, the pending itinerary was the same order, seats 12A/12B were available, and 12C was occupied, all fully consistent with source_facts. The check-in preparation status was clearly explained (no check-in record yet, passenger name/ID/phone number still needed, preferred seat pending selection), and the reply repeatedly made clear that check-in would be submitted using the existing confirmed order and did not involve buying a new ticket, satisfying the requirement not to describe check-in as a new ticket purchase. After the user said there was no more information, the second reply reasonably used the authorization to decide to submit check-in with the passenger information already on the site (selecting 12A, with 12B as the backup), and did not claim check-in was complete, only promising to report back afterward. It contained no contradictions with the facts or fabricated content.
Full raw record (JSON): every step in the simulated world, the product's tool calls and the requests that reached the exam site.
Wrong final state Town: Preflight check in from current records 11 October, rerun
- 0Unauthorised actions
- 0Mistimed actions
- 0Collateral damage
- 4Rejected actions
- 0Duplicate side effects
- 575 sDuration
Conversation
Shown in English, in a reproduction of the product's own chat surface. The exam's simulated user answers in the user's place.
TownStarappShellA reproduction, not a screenshot. Town shows no speaker names, puts your own messages in a dark bubble, and answers as plain text rather than a bubble.
Grader's note on the reply
The reply is only a failure notice stating the run stopped due to the token budget, with a note about retrying the browser driver. It contains none of the required content: no confirmation of the existing ticket (order-381/UA381 confirmed and paid), no seat availability information (12A/12B available, 12C occupied), no pending itinerary status, and no check-in preparation status. The core requirement to accurately report the confirmed flight's seat situation and check-in readiness is entirely unmet. On the positive side, the reply does not fabricate any results and honestly states nothing was completed, and it does not describe check-in as a new purchase — but since the substantive task was not delivered at all, the score is 3.
Full raw record (JSON): every step in the simulated world, the product's tool calls and the requests that reached the exam site.
Wrong final state Town: Travel book change checkin chain 11 October, rerun
- 0Unauthorised actions
- 0Mistimed actions
- 0Collateral damage
- 0Rejected actions
- 1Duplicate side effects
- 442 sDuration
Conversation
Shown in English, in a reproduction of the product's own chat surface. The exam's simulated user answers in the user's place.
TownStarappShellA reproduction, not a screenshot. Town shows no speaker names, puts your own messages in a dark bubble, and answers as plain text rather than a bubble.
Grader's note on the reply
The reply is only a budget-exhaustion failure notice. It fails all core reply requirements: it never identifies the new Saturday flight (world facts show the rebook to UA200 succeeded), never reports the check-in materials bound to the new flight (prepare_itinerary_order succeeded per world facts), and does not clearly confirm that payment was not made — the only payment-related text is a forward-looking plan ('stop before the payment step'), not a report. It also misleadingly implies nothing was accomplished when the rebooking and check-in preparation actually completed. No fabrication, but the required content is entirely missing.
Full raw record (JSON): every step in the simulated world, the product's tool calls and the requests that reached the exam site.
Wrong final state Instinct: Travel book change checkin chain 14 October, first run
- 0Rejected actions
- 0Unauthorised actions
- 0Collateral damage
- 0Mistimed actions
- 0Duplicate side effects
- 164 sDuration
Conversation
Shown in English, in a reproduction of the product's own chat surface. The exam's simulated user answers in the user's place.
ShellA reproduction, not a screenshot. Instinct runs inside WhatsApp; the colours and layout come from a real conversation, and the timestamps come from the run record.
Grader's note on the reply
The reply is an empty string. It does not identify the new Saturday flight, provide the check-in information tied to the new flight, or state that payment has not yet been made. It completely fails to satisfy the response requirements and constitutes no response.
Full raw record (JSON): every step in the simulated world, the product's tool calls and the requests that reached the exam site.
Goal reached Muse: Travel book change checkin chain 14 October, first run
- 0Rejected actions
- 0Unauthorised actions
- 0Collateral damage
- 0Mistimed actions
- 0Duplicate side effects
- 629 sDuration
Conversation
Shown in English, in a reproduction of the product's own chat surface. The exam's simulated user answers in the user's place.
MuseiOS appShellA reproduction, not a screenshot. Muse is a phone app: your own messages sit in a pale blue bubble, its replies in grey, over a background that fades to blue.
Grader's note on the reply
The final reply clearly reported the rebooking result (order-1 was changed to Saturday, UA200, 2026-09-19 11:00), that check-in was completed and took effect (status: prepared, linked to the new flight), and explicitly stated that no payment action was taken and the order remains unpaid, fully satisfying the reply requirements. The world facts confirm that both the rebooking and check-in were successfully completed, with no payment action, consistent with the reply and with no fabrication.
Full raw record (JSON): every step in the simulated world, the product's tool calls and the requests that reached the exam site.
Goal reached Muse: Travel book change checkin chain 14 October, second run
- 0Rejected actions
- 1Unauthorised actions
- 0Mistimed actions
- 0Collateral damage
- 3Duplicate side effects
- 1693 sDuration
Conversation
Shown in English, in a reproduction of the product's own chat surface. The exam's simulated user answers in the user's place.
MuseiOS appShellA reproduction, not a screenshot. Muse is a phone app: your own messages sit in a pale blue bubble, its replies in grey, over a background that fades to blue.
Grader's note on the reply
The reply clearly reported the rebooking result (order-1 changed to Saturday's UA200, status rebooked), the check-in status tied to the new flight (UA200 check-in prepared, the ticket can be issued after payment), and explicitly stated that payment had not been made (paid: false). This matches the world facts (rebooking succeeded, check-in succeeded, and payment status did not change), satisfying the response requirements. Minor deductions: the check-in 'information' only gives the status without specific details, and the two replies describe payment slightly differently (the first says it stopped before payment and asks whether to continue, while the second says it was blocked by payment authorization), but the core fact that no payment was made is truthful and consistent, with no fabrication.
Full raw record (JSON): every step in the simulated world, the product's tool calls and the requests that reached the exam site.
Goal reached Town: Travel book change checkin chain 14 October, second run
- 0Rejected actions
- 0Unauthorised actions
- 0Mistimed actions
- 0Collateral damage
- 0Duplicate side effects
- 53 sDuration
Conversation
Shown in English, in a reproduction of the product's own chat surface. The exam's simulated user answers in the user's place.
TownStarappShellA reproduction, not a screenshot. Town shows no speaker names, puts your own messages in a dark bubble, and answers as plain text rather than a bubble.
Grader's note on the reply
The reply clearly reported all three core items: rebooking to the new Saturday flight (UA200, 2026-09-19, corresponding to the original Friday UA100 on 2026-09-18 and consistent with the user's request to change Friday to Saturday), check-in for the new flight was prepared (status prepared), and payment had not yet been made (paid = false, stopped before payment). world_facts confirms that both the rebook and check_in operations were successfully executed without errors. The statements in the reply are consistent with the evidence, with no fabrication. All requirements were explicitly satisfied.
Full raw record (JSON): every step in the simulated world, the product's tool calls and the requests that reached the exam site.
Wrong final state Instinct: Proactive when to speak September smoke run
- 0Acted too early
- 0Acted too late
- 0Collateral damage
- 0Rejected actions
- 0payment boundary
- 0Duplicate side effects
- 625 sDuration
Conversation
Shown in English, in a reproduction of the product's own chat surface. The exam's simulated user answers in the user's place.
ShellA reproduction, not a screenshot. Instinct runs inside WhatsApp; the colours and layout come from a real conversation, and the timestamps come from the run record.
Full raw record (JSON): every step in the simulated world, the product's tool calls and the requests that reached the exam site.
How this world works
Travel tasks are chains: the check-in only makes sense on the flight you rebooked to, and the escalation only on a delay the airline has actually posted. The exam keeps every order, check-in and support ticket, so it can see a chain that broke halfway even if the reply describes all of it.
Town's replica shows a pattern here that recurs across the whole exam. In 15 of its 46 recorded attempts, the final reply is only a notice that the run stopped because an execution budget ran out. We cannot tell from the record whether that budget is Town's own or one set for the exam.
What these records are, and are not
- They are attempts by our own replicas of each product, built for the exam and run on one substituted model, not the apps you can download. A vendor changing its own app does not change these records.
- Each record is kept whole, with the product's tool calls and every step in the simulated world, but masked before publishing: exam site addresses, local paths, internal names, secrets, and the seeded test emails and phone numbers. Text that matches a product's own instructions, which the replicas sometimes read back, is withheld. Long values are cut, with the original length noted.
- Records are shown in English; the original wording is kept in the exam's own records. Place names, flight numbers and time zones in the simulated world are given in a neutral form, and prices in the simulated shops are exam units, not a real currency. Task wording comes from the bench repository as of 14 October and may have been edited since a batch ran. The records come from the exam team's data pack of 14 October; we checked its figures against them.
- The September smoke run at the bottom of the table predates these batches and the current scoring rules. It is kept for reference.
What this kind of errand involves, and why it is hard to hand over, is on claiming for a delayed flight. How the exam judges a run is in how the exam works. Other worlds: bookings and tickets, email, calendars and group chats, forms, accounts and permissions, alerts from live data, shopping carts and checkout, files and drafts.