How to test AI agents on real websites: record and replay
By Fitz Zhang · 12 November 2026
How to test AI agents fairly comes down to one question: what does the agent touch when it works? v1 of this exam answered with simulated worlds, and every result on this site so far describes our replicas of each product's runtime running on one substituted model.
v2, the next exam on the same bench, answers differently. A task runs on real websites captured as a network-level recording and replayed offline, so the agent sees the site itself, pop-ups and all, while no request leaves the machine. One task is published in the open repository, and there are no v2 results yet.
- Mock environments are too clean and too expensive to clone; live sites rot, need accounts and drift between runs. Recording the real site once and replaying it offline keeps the realism and removes the drift.
- Replay matches each request to the recording by rules, not by exact bytes, so an agent can reach the same page through a different path or differently worded search.
- The first published task asks an agent to find the cheapest US store for Sony's WH-1000XM5 headphones, graded on a checklist of 12 items with two must-pass checks.
- Nothing reaches a live site during a run, not even on a miss: the proxy's DNS points at a dead address, so a request that slips past the plugin still cannot resolve a host.
- No v2 results exist yet. When they appear, they will be labelled separately from v1 on the results page, with rates, intervals and task counts, never a combined score or a ranking.
Why testing AI agents on mock sites or live sites both fails
Mock sitesStressWeb, 10 generated sites
54.3% average checkpoint pass rate in a clean environment
- Too clean: no cookie popups, CAPTCHAs or rate limits
- Expensive to clone: a sliver of Amazon in 24 hours
Live siteslive benchmarks
- Real accounts to create, keep logged in and lose to bans
- Drift: a changed score can mean the site changed, not the agent
The requirements the method has to meet. Sources: StressWeb · WebCanvas / Mind2Web-Live · Online-Mind2Web · WebVoyager. Same figures as the text.
Testing AI agents against a copy of a website flatters them. Benchmarks such as WebArena, REAL, Westworld, MyPCBench, PinchBench and VeriEnv hand the agent a simulated environment: sometimes self-hosted open-source apps like Postmill for a forum, Magento for a store or GitLab for code hosting, sometimes a site an agent rebuilt from the real one.
The copies come out cleaner than the web they imitate. Browserbase lists what the real web has and clones lack: cookie popups, CAPTCHAs, iframes, rate limits and layouts that change overnight. StressWeb measured what that cleanliness hides: across 10 generated websites, the average checkpoint pass rate was 54.3% in a clean environment, 43.0% once pop-ups appeared, and 26.2% when the site's interactions were quietly remapped.
Cloning is also expensive. Our own write-up of the attempt describes an agent spending 24 hours reverse-engineering Amazon and reproducing only a sliver of it.
Testing AI agents on live sites fixes the realism and breaks everything else. ClawBench, WebVoyager and Amazon-Bench run tasks on real websites and intercept the final request before an order is placed. That approach brings three problems of its own:
- Tasks rot. When the WebCanvas team built Mind2Web-Live, 96 of the 780 Mind2Web tasks they checked (12%) had already expired. By early 2025 the Online-Mind2Web authors sampled 650 tasks and found 47% invalid or with outdated answers. Only 22.3% of WebVoyager's tasks have answers stable enough to be marked golden. OSWorld-Verified took a team of roughly ten people about two months to work through 300+ issues.
- Real accounts are required. Someone has to create them, keep them logged in, and absorb the bans when a site flags them.
- Results drift. Prices, stock and layouts move between runs, so a changed score cannot tell you whether the agent changed or the website did.
From v1 simulated worlds to v2 recorded real websites
The v1 exam runs errands in stateful simulated worlds, on our replicas of each product's runtime and one substituted model (GLM 5.3 flash). Its results are published as rates with 95% intervals and task counts, with no combined score and no ranking. That design tells you how the replicas behave; it cannot tell you how a product does on the web people actually use.
v2 keeps the bench and changes the environment. A task is a recording of the real websites it needs, replayed offline so the world the agent sees is the site itself.
When v2 results are published they will follow the same rules as v1: rates with intervals and task counts, no combined score, no ranking, and a request log underneath every claim. The results page will label them separately, the same way September and October v1 results are kept separate now. There are no v2 results today.
Record and replay: how the v2 exam sees real websites
tasks/<id>/ task.yaml recording/ rules/ grading.yaml sites/ (shared rules)
Pick a request and watch where it goes.
Choose a request above.
replay.jsonl match type: –
The method, implemented in the public repository, is a record and replay loop in three parts. The figure below shows the shape of it; the details follow.
First, a person does the task once by hand in a dedicated Chrome profile while a recorder listens over the Chrome DevTools Protocol. Every request and response is stored as a WARC recording, together with a screen recording and a page timeline. Cookies, credentials and personal data are stripped before anything is committed, and an audit checks for leftovers and bot-check pages.
Second, the recording becomes a task in tasks/<id>/. The checklist lives in grading.yaml, matching rules in rules/ (site-wide ones in a shared sites/ folder), and recording/ holds the WARC bodies with a request index and page records. During replay every request is reduced to a key built from the method, host, normalized path, the query parameters that matter and the body fields that matter.
Third, the agent takes the exam through a whistle proxy and sees real URLs and real pages, HTTPS included. Pages see the recording-time clock, so a "deal ends in 2 hours" banner behaves as it did that day. Every request lands in replay.jsonl tagged by how it was answered: exact, fuzzy, search, blocked or miss.
How replay matches a request to the recording
Illustrative Amazon URL (placeholder ASIN B0EXAMPLE1). Toggle the rule sets to see which parameters get dropped.
Resulting match key:
GET amazon.com /Sony-WH-1000XM5-Headphones/dp/B0EXAMPLE1 ?ref=…&qid=…&sr=…&pd_rd_w=…&th=1API requests matched in the Amazon-task self-check, as described in the text. The URL is illustrative; the counts are real.
Open the same product page twice and the requests are never identical, so matching on exact bytes would fail almost everything. The replay rules decide which parameters identify content and which are noise. Generic rules drop t, ts, timestamp, utm_*, fbclid, gclid and any parameter with "session" in the name; Amazon adds ref, qid, sr and pd_rd_* on top.
Page identity comes from the page itself: every variant of an Amazon product URL collapses to /dp/{ASIN}. Searches match on how similar the words are rather than the exact string. State-changing requests go the other way and match only on the fields that matter, like item ID and quantity, so a different item is a miss.
Most rules are inferred from the recording itself, from values that change between loads without changing the response, canonical link IDs and near-empty beacon responses. Unmatched requests go to a program that compares them with recorded ones and adds a rule; what it cannot resolve goes to an agent, and hand-written rules win over generated ones.
On the Amazon task, generic rules alone matched 52 of 159 API requests in the self-check; with inferred rules it was 42 of 42. A differently worded search that 404'd under generic rules found the product and read its $199.00 price.
The first task on recorded real websites
Where agents wandered
- Google searches → Walmart, Costco, Target, eBay
- Price-history lookup → Pangoly → merchant link to Newegg
- Every one came back 404 under the first recording
Still open
- Bot checks while recording: Walmart, B&H, Crutchfield, CamelCamelCamel
- Costco stops at its ZIP-code prompt
- Target's price and fulfillment widgets never captured
Each run prints a coverage report of unanswered pages, which doubles as the to-do list for the next recording.
The one published task, sony-xm5-price-compare, reads like something a person would type: find the cheapest US store for Sony's WH-1000XM5 headphones. Its checklist in grading.yaml has 12 items and two must-pass checks: the agent must name the right model (the WH-1000XM5 headphones, not the WF-1000XM5 earbuds) and report availability correctly.
The remaining items ask whether it looked beyond one store, counted shipping and pickup, checked member discounts, cashback, card offers and promo codes, looked at price history, and asked for what it needed but lacked.
The first recording, captured on October 27, held 6,805 request-and-response pairs, and it was not enough. Agents shop differently from the person who recorded: they searched Google and landed on Walmart, Costco, Target and eBay, and one price-history lookup reached Pangoly, whose merchant link goes on to Newegg.
Every one of those pages answered 404 in replay, which said nothing about the agents and everything about holes in the recording. A second pass on October 30 followed the route the gaps pointed to and added 2,573 exchanges and 50 pages, bringing the task to 9,378 exchanges across 127 pages. Each run now prints a coverage report of unanswered pages and calls, which doubles as the to-do list for the next recording.
Some holes stay open. Walmart, B&H, Crutchfield and CamelCamelCamel showed bot checks even while recording, so their challenge pages are excluded. Costco stops at its ZIP-code prompt, and Target's product page replays while its price and fulfillment widgets load in batches never captured.
Grading a replayed run
extractAn LLM pulls fields out of the text answer. No score of its own; it feeds the other layers.
price ("lowest price found, without shipping") and store ("the website selling at that price").requireThe program compares fields to expected values. Scores 0 if any entry fails.
checksYes/no questions an LLM answers about the answer. required: true scores 0 on "no"; weighted questions give partial credit.the Sony task uses this layer
traceRules on the requests the agent made in the replay. Any failed entry scores 0.
no_order fails the run on a POST to a place-order URL; a companion rule requires the add-to-cart request to be a hit from the recording, not a miss.Everything shown is in the text of this page or in docs/grading.md in the repository.
A personal agent answers in text, so a task's grading.yaml can score the run from two directions at once: what the answer says and what the agent actually did in the replay. The format has four layers, shown in the figure below.
In extract, an LLM pulls fields out of the agent's text answer, such as the lowest price found and the store selling at that price. In require, the program compares those fields against the expected values, and any failure scores the run 0. In checks, an LLM answers yes/no questions about the answer; required: true questions score 0 on a "no", weighted ones give partial credit. In trace, rules run against the requests the agent made in the replay.
The format's own example is a no_order rule that fails the run if the agent POSTs to a place-order URL, beside a rule requiring that adding to cart was answered by a hit from the recording rather than a miss. The Sony task uses only the checks layer, its 12 questions.
The grader is Claude, and the run produces result.json and a readable report.html with the score, each item's result and reason, and every page that came back 404. The request log is kept with every run, so a v2 result can be checked against the requests the agent actually made, which is the same standard our grading checks apply to v1.
What record and replay testing cannot cover yet
A recording freezes the web at one moment, which is the point, and it costs real coverage. Carts do no arithmetic: adding the item added during recording works, anything else misses, and the cart page shows the state the recording ended in.
Recordings from different days mix. In the Sony task the second capture came about 2.7 days after the first, and the earlier copy wins when both recorded the same request, so prices on different pages can come from different moments. Sites that block bots while recording stay out of the task entirely, and a task that plays out over days, like "tell me when this price drops," cannot fit inside one frozen recording.
There is also what the exam format itself does not cover. The first task was recorded without signing in to any account, so anything behind a login is untested. These limits are why v2 results will be published next to v1's, not merged into them.
How to test AI agents against the recording yourself
The replay engine, the first task, its recording and its grading checklist are all in the Personal Agent Benchmark repository. If you run your own recording setup, the details worth getting right are the ones this exam had to learn the hard way:
- Strip cookies and credentials at recording time, and audit for leftovers before anything is committed.
- Decide which request parameters identify content and which are noise, and let the recording teach you most of it.
- Never fall through to the live site on a miss; answer locally and point the proxy's DNS somewhere dead so nothing leaks.
- Freeze the page clock to the recording time, or countdowns and deal timers will lie.
- Treat a recording-gap 404 as a to-do for the next recording rather than an agent failure, and print the list on every run.
It needs Node 22+ and Chrome:
``bash npm install npm run build node bin/pab.js list node bin/pab.js replay sony-xm5-price-compare ``
replay unpacks the recording, starts the proxy and opens a Chrome window on the first recorded page. To connect your own agent, drive that Chrome over CDP or bring your own browser pointed at the proxy:
``bash node bin/pab.js replay <task> --cdp-port 9333 # agent drives this Chrome over CDP node bin/pab.js replay <task> --no-browser # proxy only; bring your own browser ``
A browser of your own needs --proxy-server=http://127.0.0.1:<port> --ignore-certificate-errors --disable-quic. To score a run, hand the grader the replay log and the agent's final answer:
``bash node bin/pab.js grade <task> --log .cache/logs/replay-<task>-<time>.jsonl --answer answer.md ``
That writes result.json and report.html to .cache/runs/, with the score, each graded item's reason and the pages that missed. The repo's demo README walks through a minimal agent (pi with browser-use) run against a task replay, including a report of what the recording missed.
The engine is also published on npm as personal-agent-benchmark, so npm install personal-agent-benchmark@0.2.0 gives you the pab command and the replay modules without a checkout; tasks are not part of the package. For how this exam compares with the public suites, the benchmarks page covers WebArena, OSWorld, WebVoyager and the rest. The longer story of building the recorder, including the failed attempt to clone Amazon for 24 hours, is on Medium.
FAQ
Are there any v2 results yet? No. One task, the Sony headphone price comparison, is published with its recording and grading checklist, and no product has a published v2 score. Results will appear on the results page labelled separately from v1.
Does the agent ever touch the live website? No. All traffic goes through a local replay proxy that only reads the recording, and the proxy's DNS points at a dead address, so even a request that slipped past the replay plugin could not resolve a host. A page the recording cannot answer gets that site's own captured 404 page.
How is this different from WebArena? WebArena hosts working copies of four kinds of sites and scores tasks inside them, so the environment is controlled but cleaner than the live web. v2 records the real sites once and replays them, so the agent sees the actual pages, pop-ups and prices, and every run sees the same ones.
Is the code open source, and under what licence? The replay engine, the task format and the first task are public on GitHub. The repository's README says "Not decided yet" in its License section, so no licence has been granted so far.
Can I add a task or a recording? The repository documents the task format in docs/tasks.md and accepts tasks through pull requests, after a replay self-check and review. Recordings must pass an audit for leftover cookies, credentials and bot-check pages before they are committed.
Fitz Zhang, Engineer, Personal Agent Bench
Builds the v2 record-and-replay exam on Personal Agent Bench. Previously a frontend engineer at ByteDance and Tencent Docs.