Also in简体中文한국어

How to compare AI agents: the six questions we ask

How to compare AI agents comes down to the questions you ask, because the questions decide what you find out. Count features and you learn what each marketing department chose to put on a page.

• The six questions: where does it run, how do tools reach the model, where does your memory live, what do you get to approve, how do you reach it and what can start it without you, and what could nobody establish.
• They were written after we took the products apart, not before. They mark the places the four turned out to differ, rather than a checklist we arrived with.
• Nothing is weighted and there is no combined number, because combining them means deciding how much each matters to you.
• The last question, what we failed to establish, is answered for every product, and those answers are not short.

How to compare AI agents with the same six questions

If you are choosing an agent rather than reading about ours, this is the short version. Ask each candidate these, in this order, and write down what you actually saw rather than what the product page says.

1. Where does it run? On a server you never see, in a machine of your own in the cloud, or as a process on your laptop that can see your messages and screen.
2. How do tools reach the model? All at once, a group at a time, or behind a separate list of what is allowed.
3. Where does your memory live? In a file you can open and edit, or in a store you can only reach through the product.
4. What do you get to approve? Which actions stop and wait for you, what exactly you are approving, and whether you can take back what it knows.
5. How do you reach it, and what can start it without you? The surfaces you type into, and the clocks, inboxes and calendars that can wake it.
6. What could you not find out? Write the gaps down as answers. An empty cell that means "didn't look" and one that means "nothing there" should not look the same.

Two practical checks sit outside that list: whether you can get in at all, since some of these are invite-only or waitlisted, and what it costs today, read from the vendor's own pricing page on the day you look. Town's pricing page, for example, starts with a free plan, while Instinct showed no price on its homepage or in its terms when we checked on 12 October 2026.

For example, Instinct’s examined messaging route has exact-plan approval, a leased browser and an unknown main-agent host. Its September replica rate cannot establish whether it will complete your own booking. Before choosing, write down the desired final state, permitted accounts, required availability and a dated price quote. Then run a small authorised trial and retain the outcome and approval record. This checklist complements the six architecture questions; it does not combine them into a score.

On our scorecard the same six questions are the rows, summarised below:

Read across the scorecard to compare one axis, and down a column to inspect one product. The teardowns provide the supporting observations and limits.

Where a personal agent runs, and why it decides the rest

The question isn't "is it local or cloud", because none of these four runs a model on your machine and a yes/no would make them look identical. The question is what sits between the model and your data, and how much of your hardware is inside the blast radius when something goes wrong.

Four different answers turn up. A request assembled entirely server-side means the instructions shaping the agent's behaviour never reach you, so you can't inspect them and neither can anything else on your machine. A per-user Linux VM means the request is built somewhere that is, in a meaningful sense, yours. A sandbox can isolate local execution; actions through connected accounts can still have lasting effects. A native desktop process additionally exposes whatever local permissions it receives.

That last arrangement is worth sitting with. The model isn't on your laptop; a process that can see a great deal of your life is, and it does what the remote model tells it. Execution location decides what an agent can reach, what a poisoned web page could talk it into doing, and what evidence exists afterwards.

How tools reach a personal agent's model

A personal agent is a model plus the set of things it's allowed to do. This axis asks how that set gets in front of it and who chose.

The interesting distinction is between what a product can do and what it tells the model it can do, at the moment it's deciding. Some products push everything into context at once. Some load definitions a group at a time, only when a group becomes relevant. Some separate the question of permission from the question of capability entirely, so there's a list of what's allowed that carries no detail about how any of it works.

That separation matters most to you, because it makes "what is this thing allowed to touch" a question with a short answer. When a definition enters the context is not a detail for implementers only. Loading a whole catalogue consumes context even for tools the agent never calls. That overhead does not establish what a consumer pays under a free, flat-rate or usage-based plan. Everything in the context is also something a hostile instruction could try to reach for, so a narrow set is a narrower target. The four products here reached four different conclusions about it.

Tool counts appear on the scorecard and they come with a warning attached every time. A catalogue number counts what exists somewhere in the product. What you actually want is how many tools the assistant in front of you can reach after you've connected only the accounts you were willing to connect. No product on this bench publishes that number for your accounts. The closest is Town, where we saw its main assistant offered 177 tools out of a catalogue of 1105, which is still a figure about the product rather than about your setup.

Where your memory lives

Every one of these products remembers things about you. They disagree completely about what form that takes and who can look at it.

The axis reduces to one practical question: if this thing believes something wrong about you, how do you fix it? On this bench the honest answers run from "open the file and edit the line" to "go through the product's interface or go without". Markdown in a home directory and a server-side store you can only query are not two implementations of the same feature. They're different relationships with your own data.

A second question rides along with the first. What's kept separately from everything else? A credential store built to fill a form without revealing what it filled is a different piece of engineering from a general notes file, and a product that has both has thought about the problem.

We describe the shape of each store, never what's in it.

What you get to approve before an AI agent acts

This is the question most comparisons skip, and it is the one that decides whether you can leave an agent alone with your card. It has two halves: what the agent stops and asks you before it does something, and whether you can take back what it has learned about you.

The four products answer the first half in four different shapes. Pine asks for confirmation before a billing action and has two separate kinds of it: one to start the task, with its own expiry, and an identity confirmation when the company on the phone wants to hear from the account holder. Instinct shows an approval card with three answers rather than two, approve, deny or dismiss, and the approval binds to the exact text shown, so a rewritten message needs a new approval. Muse routes every permission and approval, and anything leaving the machine, through a single gatekeeper, with an approvals tab next to the activity feed. Meta's own security write-up names that component Sentinel and calls it the sole permission authority for connector actions and network egress. Town treats approval as configuration rather than a prompt: the run mode, the account scope, the tool list and the approval mode decide whether a task may start and whether it may write anywhere outside.

None of these is the right answer for everyone. A prompt before each risky step suits someone who wants to watch; configuration set once suits someone who wants routines to run at night. What you should check, for any agent, is what exactly you are approving. An approval tied to the precise text that will be sent is a different promise from one tied to the request that produced it.

The second half is deletion. Muse's runtime exposes file-removal and forget capabilities, but personal access was not verified. Town's memory has add, update and delete tools. Instinct documents what deletion covers. Pine has no file holding what it knows, so there is nothing for you to open and remove. Our own exam takes the same line on the first half: stepping over a payment, a signature or a final submission without the user's say-so is counted as an unauthorised action on its own line, as how the exam works explains.

How you reach it, and what can start it without you

The surface list is the boring half of this axis. Web, desktop, a messaging app, a phone call, an API other people's agents can call.

The half that matters is the trigger surface. At least two of the four products here, Town and Instinct, can begin work with no human typing anything: a clock reaching a time, an email arriving, a calendar entry approaching, a document changing. Those are the two whose teardowns document it; for Pine and Muse our teardowns do not settle the question either way. That's the feature that makes an agent useful while you're asleep and it's also the one that makes it addressable by people who aren't you.

An agent whose trigger surface includes an inbox can be spoken to by any stranger who knows your email address. Whatever check sits between "something happened" and "act on it" is therefore load-bearing in a way no feature comparison captures, and it's why this axis records what can start a run rather than only what you can type into.

What we could not establish

Most comparisons treat a gap in their knowledge as something to route around. Here it's the last axis and it's filled in for every product.

It's there for two reasons. The obvious one is honesty: a bench that only reports what it managed to find out is a bench you can't calibrate, because you've no idea whether an empty cell means "nothing to report" or "we didn't look". The less obvious one is that these entries are informative in their own right. How much of a product is visible to a determined outsider is a real property of that product, and four columns of failures tell you which companies have decided that opacity is a feature.

The entries are specific rather than gestural. Not "we don't know much about its prompt" but how many attempts were made and what came back. Not "memory is unclear" but which part of the structure surfaced and which didn't.

Why personal agents still get no combined number

The first batch on the scorecard prints completion rates with 95% intervals and scored-task counts, ordered by sample size rather than rate. Town’s unreconciled figure remains withheld. These are adapted-runtime exam measurements with unpublished batch evidence. The October batch beside them is different on that point: every attempt behind it has a record, and we checked its figures against those records. Recorded runs document individual attempts, and the exam method explains how outcomes are checked. None of these measurements is combined into one product score.

One competing scorecard, Assistant Benchmark, is marked by one person from real use, each category out of 10, with untested cells left blank. When we read it on 22 October 2026, its general board, last updated 19 October, gave an overall score to 29 assistants across up to 15 scored categories. That's a fair way to publish evidence. It then rolls each product into one overall number and a rank, and that last step is the one this scorecard won't take, for the reasons above. The benchmarks page sets it next to a second consumer scorecard that orders the same two products the other way round.

Three things this scorecard deliberately doesn't measure

Price isn't a row. Pricing changes faster than a comparison page can track, and a stale price looks authoritative while being wrong. Current figures live, dated, on each product page.

Output quality isn't a row either. Whether an agent writes a decent email is a judgement that varies by task, and scoring it tends to measure whoever does the scoring. Runtime facts are checkable by anyone with an account, so that's what gets published.

Our own products aren't on the table and won't be. Zentor builds agent software, the conflict is real, and staying off the bench is the only handling of it that survives contact with a sceptical reader. Who runs this bench sets out the rest of it, including how the site is paid for.

The table covers the runtime teardowns only. The separate open-source agent comparison records pinned source findings without adding exam scores. About Personal Agent Bench explains how the observations, external reports and exam measurements are separated, and where independent verification is not possible.

The separate 114-task comparison includes Town. Its older builds, filtering and unreleased transcripts limit interpretation; it does not replace the main table.

FAQ

How do you compare AI agents without scoring them? By asking every product the same six questions and publishing the answers side by side. A comparison needs identical questions and honest answers; it doesn't need a total at the bottom, and the total is the part that hides the reasoning.

Why are the axes weighted equally? Is there a single score? They aren't weighted at all, and there is no single score. Weighting would mean deciding how much "the model runs server-side" matters relative to "memory is a file you can open", and that depends entirely on your situation. Anybody publishing a combined score has made that decision for you without saying so.

Where did the six axes come from? From the teardowns. We took four products apart first and built the axes out of what turned out to be genuinely different between them, which is why they're oddly specific rather than the categories a feature table would use.

How do you test how well AI agents perform? In a separate exam of everyday errands. The September rates came from the team running the exam and have no public batch record behind them. In the 11 October batch each of the four was given the same 20 tasks once (Pine has no figure from it, because of an exam fault), and we checked the figures against the record of every attempt. Both are on the scorecard with their intervals, and neither is added to the teardown answers.

What should I look for when comparing AI agents? Where it runs, what it can reach, where its memory lives, what it asks before acting, what can start it without you, and what nobody could find out. Then check access and the current price on the vendor's own page. Teardown observations concern account-visible behaviour, so you can check them with an account of your own; where we couldn't observe something, it's in the last row instead of being smoothed over.

Comparing AI agents on evidence you can check

The method here is not clever: ask the same six questions of every product, write down what you saw, say plainly where you failed, and don't add the results into one number.

What that buys is a page you can argue with. Every cell points at observable behaviour, so a reader who disagrees has somewhere concrete to disagree. See the four teardowns the axes came out of, the scorecard itself, or what the existing agent benchmarks measure and why none of them answers this question.

Questions taken across every agent

Some of these questions get a page of their own, which asks it of every product we know of rather than one at a time.

• Run AI agents 24/7: runtime architecture and limitsWhere it runsRun AI agents 24/7: compare cloud services, local computers and self-hosting costs, with four examined runtimes and separately labelled vendor claims.
• Proactive AI agents: what starts them, and what they ask before actingHow you reach itProactive AI agents mapped onto a published research framework: what can start each product we took apart, and what it asks before it acts.
• Is your website agent ready? What personal agents do when they open itWhere it runsWhat an agent ready website needs, from our own logs and teardowns: how four personal agents open pages and what readiness checkers leave out.
• Personal Agent Protocol (Poppy): what the draft says, and who has shipped itWhat you get to approvePersonal Agent Protocol (Poppy) read from the draft itself: what poppy.json holds, why read and write scopes went, and which partners publish one.