Is your website agent ready? What personal agents do when they open it
An agent ready website, in the sense most guides mean, is one that publishes the right files: a robots.txt with AI rules, Markdown versions of pages, an llms.txt, perhaps an MCP server card. This page asks a narrower question with evidence we have and they don't. When a person sends a personal agent to your site to do something for them, what actually arrives, from where, and what does it read? We answer from two sources: the access logs of this site, and the teardowns of four personal agents we have taken apart ourselves.
- In one week of our own logs, every request from ChatGPT-User, the agent OpenAI sends when a person asks ChatGPT something, was for an HTML page. None was for the Markdown copies we publish of every page.
- The Markdown copies were read only by crawlers that build indexes. So .md files and llms.txt serve indexing, and our logs show no sign of them reaching the agent a user sent.
- Three of the four personal agents we examined can do browser work on a machine that isn't the user's. To your site that looks like a sign-in from a data centre, on a real person's behalf.
- Readiness checkers, Cloudflare's included, measure which standards a site declares. None of them sends an agent to finish a task, so a high score says nothing about whether one can.
- Whether a bot check stops these agents is the one thing we have not measured, and we say why below.
Two visitors an agent ready website has to serve
Machine traffic from AI companies is two different things, and most advice treats it as one. The first is the crawler: it visits on its own schedule, reads widely, and feeds an index or a training set. GPTBot, ClaudeBot, Amazonbot and OAI-SearchBot are in this group. The second is the agent a person sent: it visits because someone asked it something a minute ago, reads one or two pages, and goes away.
OpenAI states the difference plainly in its documentation for site owners. ChatGPT-User is the second kind: "When users ask ChatGPT or a CustomGPT a question, it may visit a web page with a ChatGPT-User agent." It adds that it "is not used for crawling the web in an automatic fashion," and that "because these actions are initiated by a user, robots.txt rules may not apply."
That second visitor is the one an agent ready website exists for. A crawler that misreads your page costs you an index entry. An agent that misreads it costs a person their errand, while they wait.
Here is what both looked like on this site. In the seven days to 20 October 2026, Cloudflare's AI Crawl Control counted about 2,100 requests from AI companies. ChatGPT-User made 610 of them, every one for an HTML page: the home page 258 times, the Instinct teardown 62, the Muse teardown 26, the results page 24. The crawlers made most of the rest.
Every page here also has a Markdown twin, which we built for agents and advertise on the page itself. Those twins were fetched 256 times in the same week, and the identifiable requests all came from indexing crawlers, led by Amazonbot (98), OAI-SearchBot (74) and ClaudeBot (49). The user-triggered agent didn't touch them. It came for the HTML, the same page a person gets.
How four personal agents open your website
ChatGPT-User fetches a page and reads it. Personal agents go further: they open a browser, sign in, fill a form and press the button. Where that browser runs decides what your site sees, and each of the four products we took apart has its own answer. The runtime details are on where it runs; this is the part a site owner needs.
| Agent | Where its browser runs | What we observed |
|---|---|---|
| Instinct | A leased cloud browser, not the user's Chrome | It can use sign-ins the user has saved. We did not identify who operates the browser. |
| Muse | In the web product, a second machine, separate from the Linux VM Meta gives each user | Meta says the agent is paused while the user takes over. Meta also says its Mac app, out since 17 September, can act in your applications; we haven't examined it. |
| Pine | On the Mac app, a separate browser agent on the user's own computer. On the cloud route, a leased remote desktop | The cloud desktop can be watched over VNC and taken over. Whether the Mac helper uses the user's sign-ins we did not establish. |
| Town | We did not observe a browser path | Its Mac app is native and reaches messages, contacts and the screen. Its web search tool doesn't say who provides it. |
So of four products, three can put a browser on your site that is not on the user's computer. Only Pine's Mac route runs where the user is, and even there the browser is a separate agent rather than the user's own window.
What a site sees when a personal agent signs in
A cloud browser carrying a real user's saved session is a strange visitor. The account is genuine and the person did ask for this. But the request comes from a cloud provider's network, often in a different city or country from anywhere that account has signed in before.
Security systems are built to notice exactly that. Our explainer on how Instinct works notes the side effect, and it fits one reported incident: a user who asked it to lower his phone bill said its attempt to sign in to his carrier set off a two-factor prompt labelled as coming from Iran. We can't tell from outside whether that attempt was harmless, and that's the point. The site couldn't either.
Three things follow for an agent ready website that has accounts.
First, an unfamiliar network on a known account is no longer good evidence of an attacker. It may be the account holder's agent. Asking the person to confirm, by a prompt or a code sent somewhere the agent can't see, serves both cases better than locking the account.
Second, one-time codes are the weak point. Two reports describe Instinct reading a login code out of a connected inbox without asking. A site that sends codes by email is, for some users, sending them to an inbox an agent can search.
Third, the last step is where people take back control. A reviewer who kept control of a Muse checkout did it by taking over the browser and typing the card in herself, as our review roundup records, and Pine's cloud desktop can be taken over too. A checkout that survives a change of hands, without restarting the session, works with how these products are actually used.
What agent readiness checkers measure
The best known checker is Cloudflare's, at isitagentready.com. Cloudflare introduced it on 17 April 2026, and searches for the phrase agent ready website only start the month after. Its checks cover discoverability (robots.txt, sitemap, Link headers), content (Markdown content negotiation), bot access control (AI bot rules, content signals, Web Bot Auth) and capabilities (API catalogs, OAuth discovery, MCP server cards and others). Commerce protocols are checked but, per the launch post, do not count towards the score. A later changelog splits it into six scores in Cloudflare's URL Scanner.
Cloudflare also ran the checks across the 200,000 most visited domains, and the launch post reports the results: 78% have a robots.txt, 4% declare AI usage preferences in it, 3.9% pass Markdown content negotiation, and MCP server cards and API catalogs together appear on fewer than 15 sites.
What the score measures is declaration. Per the launch post, "Cloudflare makes requests to it to check which standards it supports." No agent is sent to book, buy or sign in. That is a fair design for a free scanner and we are not criticising it. It means a high agent readiness score and a page an agent can actually use are different properties, and only the first is being measured.
The other checkers ranking for the term work the same way, scanning for files and headers. The two guide articles we read from Google's top ten for this phrase on 21 October 2026 were checklists built from standards and page audits; neither reported watching a real agent use a site.
Does an agent ready website need llms.txt?
Not for the visitor it exists to serve, on our evidence. This site has published llms.txt and a Markdown copy of every page since 13 October 2026. The agent people send, ChatGPT-User, has not read one of those copies in the week we measured. The crawlers have, and that is a reasonable use: an index built from clean Markdown is probably a better index.
Cloudflare's own checker doesn't test for llms.txt by default. Its launch post says it checks "whether the site correctly handles Markdown content negotiation" and does "not check for llms.txt".
So if you are choosing where to spend an afternoon, the HTML your server sends is the priority, because that is what the agent fetched. We can't tell from our logs whether ChatGPT-User runs a page's scripts before reading it, so the cautious reading is that anything an errand depends on, such as a price, a policy, opening hours or a form's fields, should be in the HTML the server returns.
Bot checks and AI agents: what we have not tested
The question site owners ask most is whether their bot protection will turn away an agent a customer sent. We built a task for exactly that. In our exam's browser tasks, one partner page checks for automated browsers before it shows a price, and the agent has to get through legitimately.
That task is not in the next baseline batch, and the reason is ours. When the exam supplies the browser, that browser can't hide that it is automated, so a product using it would fail for a reason that has nothing to do with the product. Products that bring a browser able to pass can be run on it separately, labelled as such. Until that happens we have no result, and we won't guess one.
What we can say from the architecture: the three cloud browsers above arrive from data centre networks, which is the first thing many bot systems weigh. Pine's Mac route leaves from the user's own machine. Whether that difference decides the outcome is an empirical question, and the task exists to answer it.
Making a website ready for AI agents, from what we observed
Most published checklists start from standards. This one starts from what arrived at our site and what the four agents we took apart do. It is shorter for that reason.
- Put what the errand needs in the HTML response. That is what the user's agent fetched, every time.
- Identify agents by published IP ranges, not by user agent text. OpenAI publishes the ranges ChatGPT-User comes from.
- Treat a data centre sign-in on a real account as a question, not an attack. Confirm with the person rather than locking them out.
- Assume codes sent by email may be read by an agent. If a code authorises something serious, consider whether email is the right channel for it.
- Make the final step hand-over friendly. Payment and confirmation pages that survive a person taking over a live session match how these products are used.
- Publish llms.txt and Markdown if you like. They help indexes; our logs don't show them reaching the agent a person sent.
What we couldn't establish
- Whether bot checks stop any of these agents. The task exists and has not been run with a browser that can pass it.
- Whether ChatGPT-User executes scripts. Our logs show what it requested, not how it read it.
- Who operates Instinct's cloud browser or Pine's cloud desktop. On the accounts we examined we couldn't see the provider.
- Whether Pine's Mac helper carries the user's sign-ins, and whether Town browses the web at all beyond its search tool.
- How other agents behave on sites other than ours. Our numbers are one site's week, and a site about personal agents is not a typical site.
FAQ
What is an agent ready website? Usually, a site that publishes the files and headers AI agents look for: robots.txt rules for AI bots, Markdown versions of pages, and discovery files for APIs or MCP servers. That is what readiness checkers score. Whether an agent can complete a task on the site is a separate property that no checker currently tests.
How do I check if my website is agent ready? Cloudflare's free checker at isitagentready.com reports which standards your site declares, and it is a reasonable start. Then look at your own logs for user-triggered agents such as ChatGPT-User, which tell you which pages agents actually came for, and try the errand yourself with an agent.
Do AI agents read llms.txt? On our site, the agent a person sends did not read our llms.txt or any Markdown copy in the week we measured. Indexing crawlers did. That is one site's evidence, not a rule, but it matches Cloudflare's checker leaving llms.txt out by default.
Can AI agents get past CAPTCHA and bot checks? We don't know yet. Our exam has a task for it, held out of the next batch because the browser we supply can't hide that it is automated. What we know is that three of the four agents we examined browse from cloud machines, which bot systems tend to weigh.
Why did I get a sign-in alert from another country? If you use a personal agent with a cloud browser, its sign-ins come from the cloud provider's network, which can sit far from you. One Instinct user reported a two-factor prompt labelled as coming from Iran during an errand. Check what your agent was doing before assuming the worst.
Should I block AI agents from my website? Blocking crawlers and blocking user-sent agents are different decisions. OpenAI says robots.txt rules may not apply to ChatGPT-User because a person initiated the visit. Blocking the agents your customers send means blocking those customers' errands.