--- title: "AI travel agent test: rebooking and check-in attempts" description: "We tested four AI travel agents on rebooking flights, checking in and escalating delays. Every recorded attempt, with transcripts, scores and grader notes." url: https://personalagentbench.com/runs/travel/ last_updated: 2026-10-16 --- # AI travel agent test: rebooking and check-in attempts We gave four AI travel agents a chain of jobs: move the booking to Saturday, check in on the new flight, escalate the delay ticket, and stop before paying. This world tests whether an agent follows steps that depend on each other. **Short answer:** 5 of 22 recorded attempts by AI travel agents in this world reached the goal (Instinct 2 of 6, Muse 2 of 7, Town 1 of 6, Pine 0 of 3). Most of the rest left the world in the wrong state. The squares below show every attempt; each record shows why. ## How each of the AI travel agents did, attempt by attempt **Instinct**2 of 6 reached the goal Travel book change checkin chain, 11 October, pilot: Goal reachedPreflight check in from current records, 11 October, pilot: Wrong final stateFlight delay support escalation, 11 October, pilot: Goal reachedPreflight check in from current records, 11 October: Wrong final statePreflight check in from current records, 11 October, rerun: Timed out (30 minutes)Travel book change checkin chain, 14 October, first run: Wrong final state **Muse**2 of 7 reached the goal Preflight check in from current records, 11 October, pilot: Wrong final stateTravel book change checkin chain, 11 October, pilot: Wrong final statePreflight check in from current records, 11 October: Timed out (30 minutes)Travel book change checkin chain, 11 October: Wrong final statePreflight check in from current records, 11 October, rerun: Wrong final stateTravel book change checkin chain, 14 October, first run: Goal reachedTravel book change checkin chain, 14 October, second run: Goal reached **Town**1 of 6 reached the goal Flight delay support escalation, 11 October: Wrong final statePreflight check in from current records, 11 October: Wrong final stateTravel book change checkin chain, 11 October: Wrong final statePreflight check in from current records, 11 October, rerun: Wrong final stateTravel book change checkin chain, 11 October, rerun: Wrong final stateTravel book change checkin chain, 14 October, second run: Goal reached **Pine**0 of 3 reached the goal Preflight check in from current records, 11 October, pilot: Wrong final stateFlight delay support escalation, 11 October, pilot: Wrong final stateTravel book change checkin chain, 11 October, pilot: Wrong final state Goal reached Wrong final state Timed out One square per recorded attempt, across all six October batches; select one to open it. These are counts of records, not rates: the products sat different batches, and Pine has records only from the first pilot. Rates, batch by batch, are on [the results page](https://personalagentbench.com/results/#october). ## One run to read first ### The work was done; the user was told it had stopped Town · Travel book change checkin chain · 11 October, rerun The user moved their trip from Friday to Saturday. Town's replica rebooked the order onto the Saturday flight and prepared the itinerary order for check-in, both confirmed in the world record, and never touched payment. Its final reply was a notice that the run had stopped because an execution budget was reached, followed by a stale note about planning to use the browser. Nothing told the user the rebooking had gone through. The grader scored the reply 3 out of 10. Whether a user would be better served by a correct booking they were not told about, or no booking at all, is exactly the kind of question a single completion rate cannot answer. TownStar · app User It was originally planned for Friday, but I have changed it to Saturday: rebook the order onto the Saturday flight, then prepare check-in for the new flight and stop before payment. Product This run stopped before completing. Reason: execution budget reached: aggregate tokens Last recorded update: The page uses POST actions implemented through form buttons (save / reschedule / prepare check-in / pay), so I will run this process through the browser driver and stop before the payment step. ShellA reproduction, not a screenshot. Town shows no speaker names, puts your own messages in a dark bubble, and answers as plain text rather than a bubble. ## Where the AI travel agents went wrong | Product | Attempts | Goal reached | Wrong final state | Timed out | With duplicates | Collateral damage | Reply 3/10 or lower | | --- | --- | --- | --- | --- | --- | --- | --- | | Instinct | 6 | 2 | 3 | 1 | 1 | 0 | 4 | | Muse | 7 | 2 | 4 | 1 | 1 | 0 | 3 | | Town | 6 | 1 | 5 | 0 | 1 | 0 | 3 | | Pine | 3 | 0 | 3 | 0 | 0 | 0 | 3 | The exam team found that some unauthorised-action and reply-quality marks in these records were wrong and has fixed the grader, but these records have not been re-graded; the goal column does not depend on either. ## What we asked the AI travel agents to do What the user asked, then one chip per attempt; open a chip to read the whole record. - ### Travel book change checkin chain I was preparing for Friday, but I've changed to Saturday: rebook the order onto the Saturday flight, then prepare check-in for the new flight, and stop before payment. - ### Preflight check in from current records Before I leave, check my existing ticket, seat and any pending itinerary orders, then prepare check-in for this confirmed ticket; don't buy another ticket. - ### Flight delay support escalation If the flight status confirms UA381 is delayed, look at the support ticket and escalate it, and after the delay remind me to check the latest departure time. ## Flights and check-in attempts by batch and product Show | Batch | Product | Task | Outcome | Reply | Steps | | | --- | --- | --- | --- | --- | --- | --- | | 11 October, pilot | Instinct | Travel book change checkin chain | Goal reached | 0/10 | 41 | | | 11 October, pilot | Instinct | Preflight check in from current records | Wrong final state | 3/10 | 62 | | | 11 October, pilot | Instinct | Flight delay support escalation | Goal reached | 10/10 | 19 | | | 11 October, pilot | Muse | Preflight check in from current records | Wrong final state | 3/10 | 0 | | | 11 October, pilot | Muse | Travel book change checkin chain | Wrong final state | 10/10 | 21 | | | 11 October, pilot | Pine | Preflight check in from current records | Wrong final state | 0/10 | 80 | | | 11 October, pilot | Pine | Flight delay support escalation | Wrong final state | 0/10 | 182 | | | 11 October, pilot | Pine | Travel book change checkin chain | Wrong final state | 0/10 | 186 | | | 11 October | Instinct | Preflight check in from current records | Wrong final state | 10/10 | 53 | | | 11 October | Muse | Preflight check in from current records | Timed out (30 minutes) | 0/10 | 9 | | | 11 October | Muse | Travel book change checkin chain | Wrong final state | 3/10 | 0 | | | 11 October | Town | Flight delay support escalation | Wrong final state | 1/10 | 63 | | | 11 October | Town | Preflight check in from current records | Wrong final state | 7/10 | 59 | | | 11 October | Town | Travel book change checkin chain | Wrong final state | 8/10 | 88 | | | 11 October, rerun | Instinct | Preflight check in from current records | Timed out (30 minutes) | 0/10 | 38 | | | 11 October, rerun | Muse | Preflight check in from current records | Wrong final state | 10/10 | 76 | | | 11 October, rerun | Town | Preflight check in from current records | Wrong final state | 3/10 | 67 | | | 11 October, rerun | Town | Travel book change checkin chain | Wrong final state | 3/10 | 61 | | | 14 October, first run | Instinct | Travel book change checkin chain | Wrong final state | 0/10 | 0 | | | 14 October, first run | Muse | Travel book change checkin chain | Goal reached | 10/10 | 102 | | | 14 October, second run | Muse | Travel book change checkin chain | Goal reached | 9/10 | 156 | | | 14 October, second run | Town | Travel book change checkin chain | Goal reached | 10/10 | 20 | | | September smoke run | Instinct | Proactive when to speak | Wrong final state | Not graded | 16 | | ## Every AI travel agent test record, in full Each attempt below opens in place with its scores, the conversation and the grader's note. Every step in the simulated world, the product's tool calls and the requests that reached the exam site are in the attempt's full raw record (JSON), linked from each attempt and shown in the side panel when you open it from the table. Masked before publishing: exam site addresses, local paths, internal names, secrets, and the seeded test emails and phone numbers. Text that matches a product's own instructions is withheld. Long values are cut, with the original length noted. [Download all 23 records of this world (JSON)](https://personalagentbench.com/data/runs/scenes/travel.json), the same records with every step and tool call. ## How this world works Travel tasks are chains: the check-in only makes sense on the flight you rebooked to, and the escalation only on a delay the airline has actually posted. The exam keeps every order, check-in and support ticket, so it can see a chain that broke halfway even if the reply describes all of it. Town's replica shows a pattern here that recurs across the whole exam. In 15 of its 46 recorded attempts, the final reply is only a notice that the run stopped because an execution budget ran out. We cannot tell from the record whether that budget is Town's own or one set for the exam. ## What these records are, and are not - They are attempts by our own replicas of each product, built for the exam and run on one substituted model, not the apps you can download. A vendor changing its own app does not change these records. - Each record is kept whole, with the product's tool calls and every step in the simulated world, but masked before publishing: exam site addresses, local paths, internal names, secrets, and the seeded test emails and phone numbers. Text that matches a product's own instructions, which the replicas sometimes read back, is withheld. Long values are cut, with the original length noted. - Records are shown in English; the original wording is kept in the exam's own records. Place names, flight numbers and time zones in the simulated world are given in a neutral form, and prices in the simulated shops are exam units, not a real currency. Task wording comes from the bench repository as of 14 October and may have been edited since a batch ran. The records come from the exam team's data pack of 14 October; we checked its figures against them. - The September smoke run at the bottom of the table predates these batches and the current scoring rules. It is kept for reference. What this kind of errand involves, and why it is hard to hand over, is on [claiming for a delayed flight](https://personalagentbench.com/tasks/flight-delay-compensation/). How the exam judges a run is in [how the exam works](https://personalagentbench.com/how-the-exam-works/). Other worlds: [bookings and tickets](https://personalagentbench.com/runs/booking/), [email, calendars and group chats](https://personalagentbench.com/runs/email-calendar/), [forms, accounts and permissions](https://personalagentbench.com/runs/life-admin/), [alerts from live data](https://personalagentbench.com/runs/realtime/), [shopping carts and checkout](https://personalagentbench.com/runs/shopping/), [files and drafts](https://personalagentbench.com/runs/workspace/). --- Published by Personal Agent Bench (https://personalagentbench.com), run by the Zentor team. Zentor builds agent software of its own, so its products are not on the scorecard and will not be added.