Tau bench (τ-bench): leaderboard, pass^k and versions

Tau bench is an agent benchmark from Sierra, first published in June 2024 by Shunyu Yao, Noah Shinn, Pedram Razavi and Karthik Narasimhan, that asks something harder than whether a model can complete a task: whether it can complete the same task several times running. That change of question produces one of the most uncomfortable numbers in agent evaluation, and it is why this is the benchmark on our list closest to what a personal agent is asked to do.

• An agent plays customer service inside a narrow domain, with tools and written policy, against a user simulated by another model. Success is judged on the final database state, not on what the agent said.
• Its distinctive metric is pass^k, the share of tasks an agent completes on every one of k attempts. Success once and success reliably are very different numbers.
• The original paper found state-of-the-art function-calling agents succeeding on under half the tasks, and under 25% on all eight attempts in the retail domain.
• There are now three generations, tau bench, tau2-bench and τ³-bench, and a score means little without its version and domain. On 5 November 2026 the best single-attempt score in the newest domain, banking, is 55.2%, falling to 35.1% across four attempts.
• The benchmark has re-graded its own leaderboard after finding tasks were wrong, and says plainly that old and new banking results are not comparable.

What tau bench measures, and who built it

Tau bench puts a language agent in a customer service role inside one domain, originally an airline and a retail shop. The agent gets domain-specific API tools and policy guidelines it is expected to follow. On the other side is a user, simulated by a language model, who has a goal and will not necessarily state it cleanly.

The agent has to work out what the user needs, use the tools to do it, and stay inside the policy. A refund the policy does not allow is a failure even if the user asked for it and went away happy.

Scoring compares the database at the end of the conversation with the goal state written for the task. Either the booking was changed or it was not. The same rule holds in the newer banking domain: the τ-Knowledge paper gives each of its 97 tasks a target database state and scores whether the agent's actions produce it.

Sierra sells AI agents for customer service, which explains the setting. Its research team maintains the benchmark and runs the public leaderboard at taubench.com.

The tau bench metric that makes it uncomfortable

Most benchmarks report a pass rate: run every task once, count the successes. Tau bench also reports pass^k, the share of tasks the agent completed on all k attempts.

The gap between those numbers is the whole story. On the original repository's leaderboard, the strongest airline entry, Claude 3.5 Sonnet from October 2024, scored 0.460 at pass^1 and 0.225 at pass^4. Same agent, same tasks, asked four times instead of once, and it gets all of them right less than half as often. The paper reports retail falling under 25% at pass^8.

An agent that succeeds 46% of the time is a coin flip you can supervise. An agent that succeeds every time on 22% of tasks is something else, and the difference only shows up if somebody runs the task again.

Tau bench, tau2-bench and τ³-bench: which version a score belongs to

The name now covers three generations, and quoted scores rarely say which.

The original tau bench is the 2024 paper: airline and retail, a simulated customer, scores like the 0.460 above. Tau2-bench, published by Sierra in June 2025, added a telecom domain with "dual control", where the simulated user also has tools and the agent has to talk them through actions only the user can take. That is closer to tech support than to a booking desk, and agents score noticeably worse when they have to guide a user rather than act alone.

τ³-bench, which still lives in the tau2-bench repository, adds three things. The banking domain, published as τ-knowledge, makes the agent search roughly 700 interlinked policy documents before acting. A voice mode, published as τ-voice, runs the same kinds of task over real-time audio. And more than 75 task fixes remove wrong expected actions, clarify ambiguous instructions and repair impossible constraints in the airline, retail and banking tasks.

There is also tau2-bench-Verified, a corrected version of the tau2 tasks from an Amazon team, with its own leaderboard. Its numbers, the original's and τ³-bench's are all on different task sets. A tau bench score with no version and no domain beside it cannot be compared with anything.

The tau bench leaderboard on taubench.com

The table shows the banking domain because it is the one almost every current submission reports. Read the two columns together. The leader solves 55.2% of tasks once and 35.1% on all four tries, so about a third of what it can do, it does not do reliably. When the τ-Knowledge paper came out in March 2026, the best pass^1 it reported was 25.52%, before the July grading fix and on older models, so the two figures are not a clean before and after.

The older domains are run by fewer submissions, all by Sierra on the same version and user simulator. From the same files on 16 October 2026, the best standard pass^1 scores are:

• Airline: Claude Opus 4.5 at 84.0%, falling to 70.0% at pass^4, across 8 current submissions that range down to 52.5%.
• Retail: Qwen3.5-397B-A17B at 84.4%, falling to 59.6% at pass^4, across 9 current submissions, one of them a custom setup.
• Telecom: Qwen3.5-397B-A17B at 97.8%, falling to 92.1% at pass^4, across 8 current submissions.

Those are the figures vendors usually quote, and they sit far above the 0.460 the original airline leaderboard topped out at, though on fixed tasks and with newer models. The site also keeps a separate legacy board for runs on the original v1 tasks and says those are not directly comparable with current ones.

Why repetition is the personal agent question

For a chatbot, one good answer is a good answer. For something acting on your accounts, a task that works four times out of ten is not 40% useful. It is a task you now have to check every time, which can cost more attention than doing it yourself.

The errands people want delegated are recurring by nature: the monthly cancellation, the weekly reschedule, the expense claim. Reliability is not a refinement on top of capability. For this category it is the capability, and that is why tau bench sits closest on our list to a personal agent despite testing a customer service one.

We have the same problem and have not answered it as well yet. The rates on our scorecard are single completion rates. The next baseline batch runs each of its tasks twice per product, which will show whether a product that succeeded once does it again, though two attempts is a weaker test than pass^4. Where we will differ from pass^k is in publishing the separate measurements rather than one consistency score, because an agent that fails by doing nothing and one that fails by cancelling your subscription twice are not equally bad.

The part of tau bench that deserves credit

The original repository now carries a warning at the top saying its airline and retail tasks are out of date, and points to τ³-bench.

More importantly, when the v1.0.1 release in July 2026 fixed a couple of banking tasks, the README stated that banking results from before and after the fix are not comparable, re-graded the affected leaderboard submissions, and noted that other domains were unaffected.

The uncomfortable implication applies to every benchmark, including ours. Some of those scores were wrong for a while, and everyone who cited them in that window cited something that has since changed. The only defence is to say which version produced a number and to keep old runs re-scorable, which is what the frozen task definitions in our own exam are for.

In tau bench, the simulated user is the load-bearing choice

Every benchmark that involves a conversation has to decide who is on the other side, and that decision sets a ceiling on what it can find out.

Tau bench uses a language model as the user, driven by a goal it has been given. That buys repeatability, which is the whole point: a human tester would behave differently on the second run and pass^k would mean nothing. It is the right trade for the question being asked.

The cost is realism. The simulated user follows version-specific instructions, and every row in the table above used GPT-5.2 in that role. Its patience, phrasing and interruptions are whatever that setup produces, not a sample of real customers. Keep the simulator beside the agent when you compare results.

Our own exam makes the same trade and tries to bound it. The simulated participant can only answer with facts the task declared in advance, and anything outside that list gets the same refusal every time. That does not make it human, but it does mean a run that succeeded did so on information the task contained rather than on a helpful accident.

There is a second-order effect. An agent that learns to work well with a legible, cooperative user is being optimised for a user who does not exist. The risk is not that scores are too high. It is that the skill being measured drifts away from the skill being sold.

What tau bench still cannot tell you about a personal agent

The domains are narrow and self-contained. A customer service agent inside one airline's tooling is not reaching across your email, your calendar and your bank at once, and the failures that come from touching several systems in one errand do not appear.

There is no standing memory. Each task starts clean, so a preference you expressed last month cannot be honoured or forgotten, and that is one of the main things people notice about living with a personal agent.

And it does not test the interface fighting back. The tools are APIs that work. A cancellation flow that hides its own exit, a retention offer, a confirmation code sent to a phone the agent cannot read: none of that is in scope.

How to run tau bench and read its scores

The current code is the tau2-bench repository, not the original one. Its README installs with uv sync (add --extra knowledge for banking), takes API keys in a .env file, and runs an evaluation with a command such as tau2 run --domain airline --agent-llm gpt-4.1 --user-llm gpt-4.1 --num-trials 1 --num-tasks 5. Runs call model providers and are billed by them; we did not run one for this page.

Reading a score takes four checks. Read pass^1 and pass^k together, since a submission quoting only its single-attempt figure has chosen the flattering half of its result. Check the version, because banking numbers before and after v1.0.1 are not comparable. Note the domain: airline, retail, telecom and banking behave differently, and an average hides which one carried the score. And check the user simulator, which each submission file records.

Then treat the whole thing as a measurement of rule-following under conversation, which it measures well, rather than a general statement about agent quality. The same caution applies to every benchmark on our list, and to our own scorecard.

FAQ

What is tau bench? A benchmark for tool-agent-user interaction, introduced by Sierra in June 2024. It places a language agent in a customer service role with domain-specific tools and written policies, puts a simulated user on the other side, and scores by comparing the final database state with the task's goal state.

What does pass^k mean? The share of tasks an agent completes on every one of k attempts, rather than at least once. It measures reliability rather than capability, and the gap between pass^1 and pass^k is usually large.

What is the difference between tau-bench and tau2-bench? Tau2-bench, from June 2025, keeps airline and retail and adds telecom, where the simulated user also has tools and the agent has to guide them. τ³-bench, in the same repository, adds a banking knowledge domain, a voice mode and more than 75 task fixes. Scores across versions are not comparable.

What is the best tau bench score? It depends on the domain. On taubench.com on 16 October 2026 the best standard pass^1 is 84.0% on airline, 84.4% on retail, 97.8% on telecom and 55.2% on banking. The banking figure drops to 35.1% when all four attempts must succeed.

Is the original tau bench still maintained? No. Its repository carries a notice that the airline and retail tasks are outdated and points to τ³-bench. Use the successor, and check which version any quoted score came from.

Does a good tau bench score mean a product will handle my admin? It is evidence about rule-following in a scripted domain with working tools and a cooperative simulated user. It says nothing about hidden cancellation routes, retention offers, credentials, or an errand that crosses several accounts. Those are the cases our own exam is built around, described in how the exam works.

The other benchmark pages

• GAIAEvery question arrives cold. Nothing carries over from the last one, so a system that remembers you scores exactly the same as one that doesn't.
• BrowseCompReading, not doing. Nothing is booked, cancelled, paid for or sent.
• WebArenaThe sites aren't yours and hold nothing about you. No saved cards, no order history, no account that remembers.
• OSWorldSkill, not memory. It asks whether an agent can operate the machine, never whether it knows why you'd want it to.
• AgentBenchNone of the environments is your life. No calendar, no inbox, no bill to dispute.
• Terminal-BenchAlmost nothing a personal agent does happens at a terminal.
• SWE-benchA closed problem with a written specification. An errand has neither, and it cannot be rolled back.
• AssistantBenchStill one-shot. No standing preferences, no routine that fires next Monday.