Organising work files, and the errand where finished is hardest to define

Type how to organize digital files into a search box and you get advice written for a person with a free Sunday: choose a naming convention, build a folder per year, touch every document once. Almost nobody follows it, because the work is dull and the reward arrives months later when you find something quickly and never notice why. This is the least glamorous errand on our bench and the one that shows the most about an agent. A cancelled subscription has a status field that either changed or did not. A tidy folder has nothing of the kind. The personal agents on this bench say they can do work like this on your behalf. Here is what the job actually demands, and what we measure when we make them try.

• There is no status field for tidy. Judging has to read the file tree itself, which is why the exam fingerprints structured state by sorting and hashing it.
• Reorganising means moving things, and moving things is how you break other things. Collateral damage is the dominant risk here, not failure to finish.
• A file organised into a place nobody can find is a loss wearing the costume of a success.
• An agent that renames or summarises a document it only half read is guessing. The exam is built to catch exactly that.

Which of them can actually do this

Two things that table does not say, and both matter more than the percentages.

None of those figures is specific to organising files. Each is a product's share of every task in the exam, which tells you how often it finishes what it is given, not how it handles this particular errand. The per errand runs are not done.

And a completion rate says nothing about how a product finishes. An agent that tidies your work folder by flattening a structure you depended on has completed the task and cost you a week of hunting. The rest of this page is about that gap.

Why the advice you find is written for humans

It is worth being straight about the search result you probably arrived from. Look up how to organize digital files and you get methodology for people: a date prefix, a numbering scheme, a promise to yourself about the downloads folder. Look up an AI file organiser instead and you mostly get tools that sort by metadata, meaning extension, created date, size, sometimes the words in the filename.

Neither is the claim a personal agent makes. The claim is that it reads what is inside the files and acts on the contents: this one is an invoice, that one is a contract draft, these are the same photo at different resolutions. Sorting by extension needs no understanding at all. Sorting by meaning needs the thing to have actually read the documents, and reading is the part that goes wrong silently.

That distinction is why this page exists: the difference between a tool that moves files by their properties and an agent that moves them by what they say.

What the errand really involves

Strip a filing job down and there are four distinct pieces. Products tend to be confident about the first and third and quiet about the second and fourth.

The problem with the word finished

On most errands there is a fact you can point at. A subscription is cancelled or it is not. A dispute has a case record or it does not. That fact is what our judging reads, because judging reads the world's final state and never the agent's description of what it did.

Filing has no such fact. There is no field called organised. There is a structure instead, and a structure is exactly the kind of thing that is easy to describe convincingly and hard to check by eye.

The exam handles this the same way it handles every other world. Each world holds real state, and that state gets fingerprinted: everything structured inside it is sorted and hashed. Two runs that produce the same fingerprint changed the same things. Applied to a folder, this is oddly clarifying. It means a run is judged on the tree that exists when the agent stops, down to what ended up where, rather than on a closing paragraph explaining that everything has been sorted into sensible categories.

It also makes the errand replayable. Reset the world, run a different product against the same starting mess, compare the fingerprints. That is why our tasks run in simulated worlds rather than against your real drive, and here the reasoning is easy to feel: nobody wants a benchmark that turns software loose on a real set of documents to see what survives.

Where the files have to be reachable

This is the errand where the runtime difference between these products stops being trivia.

Inside the exam things are level. Every world is reachable through a web page an agent can browse and through an API that looks like a service a product would integrate, and both read and write the same underlying record, so a product that clicks and a product that calls sit the same exam against the same facts.

Real life is not so even. Some of these runtimes reach your own machine and some only ever reach the cloud. One puts a native process on your Mac with genuine system access, the only case here where your local files are in scope at all. Another gives you a Linux machine of your own, so its working state is the closest thing on this bench to an ordinary folder you could open. Another runs in a cloud sandbox, so the files it can organise are the ones it can reach from there. And one is built around placing a phone call, a fine instrument for arguing with a company and no instrument at all for a directory tree.

None of that is a ranking. It is the question to answer before you delegate: when this product says it will organise your files, whose files does it mean. The teardowns go through where each one runs and what it can touch.

Organised into a place nobody can find

The dominant risk here is not that the agent does nothing. It is that it does too much.

The universal five measurements include whether the agent damaged something the task was watching. On a cancellation that is a corner case. Here it is the main event, because the errand consists of moving things, and every move is a chance to move something that was already where it needed to be: a shared folder another process reads from, a file whose path is written into a script, a document somebody else has open.

There is a subtler version that looks nothing like damage. The agent invents a category, files a contract under it, and writes a tidy summary. Nothing is broken, and the next time you need that contract you check the three places you would check and it is in a fourth. The folder is neater and you are worse off. A file organised into a place nobody would think to look is a loss dressed as a success, which is why our exam refuses to collapse damage into the goal column.

Duplicates belong in the same paragraph. An agent that loses track of a slow operation, retries, and copies a set of documents twice has reached the goal and left two versions that drift apart the moment anybody edits one. Every world counts duplicate side effects as a measurement of its own. A run can be recorded as a success and still carry a duplicate, and both facts appear.

The document it only half read

Our tasks can declare faults, and a shared controller fires them at a set logical time so the web page and the API both see the same breakage at the same moment. One of those faults was built for work exactly like this.

Content can come back truncated, with the truncation marked. The agent is told, in effect, that it has seen part of this document and not the rest. What it does next is the whole test. An agent that fills in the missing part from general knowledge is making it up rather than reading it, and on a filing job that is not an abstract sin. It is how a research memo gets filed as an invoice, or a draft renamed after a client it never mentions.

Summarising is where this bites hardest. Ask for a folder to be organised and you often get renaming and short descriptions thrown in, which sounds like a bonus and is a second task with a much lower ceiling on how reliably it can be checked. The exam is honest about that limit. When a task asks for a written artefact rather than a structured change, the machine can confirm that the file exists and has the declared shape, and it cannot confirm that the prose inside it is true. Those runs get labelled as structurally verified with the wording still unreviewed, and they stay out of the automated success figures until a person has read them. We would rather publish that awkward category than pretend a hash can tell you whether a summary is accurate.

What the exam actually checks

Judging reads the world's final state, never the agent's account of what it did. For a filing task that means reading the tree: what exists, where it sits, what it is called, what is no longer where it was. The agent's closing message is not evidence. That single rule removes the most common failure in agent demos, which is a fluent paragraph describing work that did not happen.

These are published side by side and never combined, because deciding how many stray copies equal one missed deadline is your call rather than ours. The full method is in how the exam works, and the reasoning behind refusing a single blended number is in how scoring works.

Questions worth asking any product that claims this

What we can and cannot tell you today

The work files world is built and it runs. The product by product results for this specific errand are not done.

We could fill this page with plausible verdicts and nobody would immediately know. That is exactly why we will not. The scorecard carries the completion rates we do have, with intervals and sample counts attached, and one product's cell is empty because the figure we were given disagreed with our own evidence. A task page claiming results it does not have would undo all of that.

One thing has changed since this page was drafted. The work files world now has a recorded attempt you can read: Instinct, one step in a little over three minutes, finishing with the world in the wrong state. Every measurement is shown separately, including the ones that came back zero. Read it as a record and not as a result. One attempt by one product is not a rate, and no batch level quality gate was written for it, so the run is not admitted to the comparison.

When the rest exist, each one gets a page showing the conversation as the product rendered it, the world trajectory, what it opened, and every measurement including the attempts that went nowhere. Until then, what this page offers is the shape of the problem, so you can judge any product's claim about it yourself. The nearest errand written up in the same detail is cancelling a subscription, which shares the proof problem and none of the collateral damage.

FAQ

Can an AI agent organise my files for me? Several products claim work of this shape, and the part worth checking is which files they mean. A tool that sorts by extension and date needs no access to the contents. An agent that files by meaning has to read the documents, which is a different permission question and a different failure mode. Ask where the agent runs before you ask how clever it is.

How is an AI file organiser different from a rules based one? A rules based organiser acts on properties: type, date, size, a pattern in the name. It is predictable and understands nothing. An agent acts on contents, which is more useful and less predictable, because a wrong reading produces a confidently misfiled document rather than an obvious error you would spot.

What is the biggest risk in letting an agent reorganise a folder? Damage, not failure. The errand is made of writes, so a run that goes wrong moves real things. The version people underestimate is the tidy one: files placed somewhere defensible that you would never look, which reads as success in a summary and costs you time for months.

Does the bench test this on my real files? No, and deliberately so. Tasks run in simulated worlds that can be reset and replayed, so two runs are comparable and anybody can check the result. Running a benchmark against real documents would mean results nobody can reproduce and genuine loss when a product misbehaves.

When will there be figures for this errand? When the runs are finished. The scorecard shows what has been measured so far across the whole exam, and this page will point at individual runs once they exist rather than summarising them into a verdict.