Tau-bench: the benchmark that asks whether an agent can do it twice

Most agent benchmarks ask whether a model can complete a task. Tau bench asks something harder and considerably more useful: whether it can complete the same task four times running. That one change of question produces the most uncomfortable number in agent evaluation, and it is the reason this is the benchmark on our list closest to what a personal agent is actually asked to do.

• It judges the database at the end of the conversation against an annotated goal state, not what the agent said it did. That is the same rule our own exam holds, arrived at independently.
• Its distinctive metric is pass^k, the share of tasks an agent completes on every one of k attempts. Success once and success reliably turn out to be very different numbers.
• The original paper reported state-of-the-art function-calling agents succeeding on under half the tasks, and under 25% consistently across eight trials in the retail domain.
• The benchmark has since re-graded its own leaderboard after finding its tasks were wrong, and says plainly that old and new results are not comparable. That is the most creditable thing on this page.

What it actually measures

Tau bench puts a language agent in a customer service role inside a narrow domain, originally airline and retail. The agent gets domain-specific API tools and a set of policy guidelines it is expected to follow. On the other side of the conversation is a user, simulated by a language model, who has a goal and will not necessarily state it cleanly.

The agent has to work out what the user needs, use the tools to do it, and stay inside the policy while doing so. A refund the policy does not allow is a failure even if the user asked for it and went away happy.

Evaluation does not read the transcript for signs of competence. It compares the state of the database at the end of the conversation with an annotated goal state. Either the booking was changed or it was not.

The metric that makes it uncomfortable

Most benchmarks report a pass rate: run every task once, count the successes. Tau bench also reports pass^k, which is the share of tasks the agent completed on all k attempts.

The gap between those two numbers is the whole story. From the original leaderboard, the strongest entry on the airline domain scored 0.460 at pass^1 and 0.225 at pass^4. The same agent, the same tasks, asked to do it four times instead of once, gets it right less than half as often. The paper reports the retail domain falling under 25% at pass^8.

An agent that succeeds 46% of the time is a coin flip you can supervise. An agent that succeeds consistently 22% of the time is something else, and the difference only shows up if somebody runs the task again.

Why repetition is the personal agent question

For a chatbot, one good answer is a good answer. For something acting on your accounts, a task that works four times out of ten is not 40% useful. It is a task you now have to check every time, which costs more attention than doing it yourself.

The errands people want delegated are recurring by nature: the monthly cancellation, the weekly reschedule, the recurring expense claim. Reliability is not a refinement on top of capability. For this category it is the capability.

This is why tau bench sits closest on our list to what a personal agent does, despite testing a customer service agent rather than a personal one. The shape of the question matches.

What our own bench took from the same problem

We landed on the same issue from the other end and handled it slightly differently. Our exam runs repetitions and reports both a best of N and a mean of N, side by side, rather than collapsing them.

The reason for showing both is that they answer different questions. Best of N tells you what the product can do. Mean of N tells you what it will do. A product whose two figures diverge sharply is one that works when you are watching.

Where we differ from pass^k is that we publish the independent measurements rather than a single consistency score, because an agent that fails four times out of five by doing nothing and one that fails by cancelling your subscription twice are not equally bad, and one number cannot hold that difference.

The part that deserves credit

The tau bench repository now carries a warning at the top telling you its own tasks are out of date, and points at a successor. The successor, τ³-bench, ships more than 75 task fixes: incorrect expected actions removed, ambiguous instructions clarified, impossible constraints fixed, missing fallback behaviours added.

More importantly, it states that results produced before and after a grading fix are not comparable, and it re-graded the affected leaderboard submissions rather than leaving them standing.

The uncomfortable implication is one every benchmark has to live with, including ours. Some of those scores were wrong for a while, and everyone who cited them in that window cited something that has since changed. The only defence is to say what version produced a number and to make old runs re-scorable, which is what the frozen task definitions in our own exam are for.

The simulated user is the load-bearing choice

Every benchmark that involves a conversation has to decide who is on the other side, and that decision quietly sets the ceiling on what the benchmark can find out.

Tau bench uses a language model as the user, driven by a goal it has been given. That buys repeatability, which is the whole point: a human tester would behave differently on the second run and the pass^k figure would mean nothing. It is the right trade for the question being asked.

What it costs is the awkwardness of real people. The simulated user states its need, responds to questions, and stays on topic. It does not change its mind halfway, mention the important constraint only after the booking is made, or reply to a clarifying question with something that does not answer it.

Our own exam makes the same trade and tries to bound it explicitly. The simulated participant can only answer with facts the task declared in advance, and anything outside that list gets the same refusal every time. It will not improvise to help an agent out of a hole. That does not make it human, and it does mean a run that succeeded did so on information the task actually contained rather than on a helpful accident.

There is a second-order effect worth naming. An agent that learns to work well with a legible, cooperative user is being optimised for a user who does not exist. The risk is not that scores are too high. It is that the skill being measured drifts away from the skill being sold.

What it still cannot tell you about a personal agent

The user is simulated. That is a reasonable engineering decision and it has a cost: a simulated user is more consistent, more patient, and more legible than the person whose subscription it is. Nobody in tau bench loses their temper, goes quiet for two days, or answers a question with a screenshot.

The domains are narrow and self-contained. A customer service agent inside one airline's tooling is not reaching across your email, your calendar and your bank at once, and the failure modes that come from touching several systems in one errand do not appear.

There is no standing memory. Each task starts clean, so a preference you expressed last month cannot be honoured or forgotten, and that is one of the main things people notice about living with a personal agent.

And it does not test the thing that happens when the interface fights back. The tools are APIs that work. A cancellation flow that hides its own exit, a retention offer, a confirmation code sent to a phone the agent cannot read: none of that is in scope.

How to read its leaderboard

Read pass^1 and pass^k together, never separately. A submission quoting only its best single-attempt figure has chosen the flattering half of its own result.

Check the version. Since the grading fix, numbers from different versions of the benchmark are not comparable, and the repository says so itself. A figure without a version attached is not a figure you can use.

Note the domain. Airline and retail behave differently and an average across them hides which one carried the score.

And treat the whole thing as a measurement of rule-following under conversation, which is what it measures well, rather than as a general statement about agent quality. The same caution applies to every benchmark on our list, and to our own scorecard.

FAQ

What is tau bench? A benchmark for tool-agent-user interaction, introduced by Sierra in 2024. It places a language agent in a customer service role with domain-specific tools and written policies, puts a simulated user on the other side, and scores by comparing the final database state with an annotated goal state.

What does pass^k mean? The share of tasks an agent completes on every one of k attempts, rather than at least once. It is a reliability measure rather than a capability measure, and the gap between pass^1 and pass^k is usually large.

Is tau bench still maintained? The original repository is not. It carries a notice that its airline and retail tasks are outdated and points to τ³-bench, which adds a banking domain, a voice evaluation mode and more than 75 task fixes. Use the successor, and check which version any quoted score came from.

Does a good tau bench score mean a product will handle my admin? It is evidence about rule-following in a scripted domain with working tools and a patient simulated user. It says nothing about hidden cancellation routes, retention offers, credentials, or an errand that crosses several accounts. Those are the cases we built our own exam around, described in how the exam works.

Why does this site cover benchmarks it does not run? Because the existing benchmarks answer real questions, and knowing which question each one answers is the fastest way to see what none of them covers. That gap is the reason our own exam exists rather than a reason to dismiss theirs.

The other benchmark pages

• GAIAEvery question arrives cold. Nothing carries over from the last one, so a system that remembers you scores exactly the same as one that doesn't.
• BrowseCompReading, not doing. Nothing is booked, cancelled, paid for or sent.
• WebArenaThe sites aren't yours and hold nothing about you. No saved cards, no order history, no account that remembers.
• AgentBenchNone of the environments is your life. No calendar, no inbox, no bill to dispute.