State of the Bots

Section 1: The Scraper Ecosystem Powering AI Applications & How Proxy Networks Work

1.1 Scraper Audit Tool & Results on UK Sites

In order to examine the sophistication of web scrapers against websites' cybersecurity tooling, TollBit developed the Scraper Audit, a tool for website owners to test 14 popular web scrapers, many of which are Europe-based companies, against their own sites. The Scraper Audit allows website owners to initiate scraping on any page to evaluate how well scrapers evade defenses, access content, and retrieval completeness.

In selecting web scrapers, we picked a few industry leaders and a couple of newer scrapers offering advanced technology, despite their company size and time in market, and have plans to add more to the tool. For each of these web scrapers, whenever available, we purchased subscriptions of their web scraping APIs to leverage their full suite of AI-driven parsing, advanced anti-bot bypassing, and premium proxy networks.

To demonstrate the utility of this tool in auditing scrapers, we conducted an evaluation of this tool across the top 20 sites based in the UK as defined by the Press Gazette.

Our test of these sites resulted in similar outcomes to the tests run for the previous edition of the report:

• Even paywalled content is not necessarily safe. In our testing of 14 of these scraping tools, we observed that some were able to scrape various paywalled articles in full.
• In fact, non-paywalled sites offered no discernible increase in protection compared to their paywalled counterparts, as scrapers successfully retrieved full content from the vast majority of target pages regardless of subscription barriers.
• In the rare instance where all scrapers failed to fully extract data — occurring in only one of the twenty sites — the failure was primarily attributed to server-side rendering implementations of the paywall. On these specific sites, the full article was not present in the initial page and required server-side authorization before the content was served.

1.2 Scraper Audit: Other Observations

Since our last bot report, we've added functionality to our Scraper Audit Tool to allow publishers to observe the suspected chain of events that occurs as a web scraper attempts to access their sites. In the tool, publishers can enter a specific URL in the Scraper Audit Tool and observe some of the strategies the web scrapers utilize in an attempt to scrape the page. In sum, we observed that scrapers often deploy various tactics to obtain access to sites, some hammering sites dozens of times with different user agents, sometimes masquerading as other bots (including Googlebot, Chrome, etc.), and others rotating IP addresses in order to attempt to trick the site's cybersecurity and allow them to gain access to scrape the page. We also observed that some of the web scrapers would trigger Google Ads bot on sites — concerning for advertisers who are paying for their ads to be seen by humans, and not bots/web scrapers.

A related note: In our last bot report, we noted that we did not observe any scrapers checking robots.txt files before attempting to access sites.

Here are some real examples of what we observed with this new functionality on publisher sites. Each of the examples below shows a single scraper instance attempting to access a publisher's URL:

1.2.1 Example 1: Masquerading as Google & IP Address Rotation

The scraper in this example appeared to have tried more than one disguise to get in. It first used User-Agent Spoofing and posed as Googlebot to gain trusted access. When that was blocked, it appeared to switch to a Chrome browser, and its apparent location shifted from Los Angeles, US, to San Sebastián de los Reyes, ES, meaning it also changed its IP address. Once it changed to a browser, it was first served 301 redirects, then kept trying until it got a 200 and pulled the content. This showcases that when one identity is refused, a scraper may just change who it claims to be and where it appears to come from and try again.

1.2.2 Example 2: Multiple Browser Impersonations & IP Address Rotation

This attempt to access a publisher's URL appeared to trigger 43 requests from a single web scraper, routed through many different IPs and locations, cycling between Chrome and Firefox identities throughout the attempt to access the site. The locations of the scrapes heavily varied: alongside US cities (San Jose, Ashburn, Clifton, Pennington), the scrapes arrived from multiple Hungarian locations (Sopron, Sümeg, Nyíregyháza) and Bahrain (Bu Kuwarah), all within seconds.

1.2.3 Example 3: Multiple Browsers & IP Rotation

This scrape request appeared to result in 32 requests hitting the URL at essentially the same time, from many different IPs across scattered US cities and cycling between Chrome and Firefox identities throughout. One scrape is arriving from a dozen-plus origins at once, and being disguised on both the network and identity side. The status column is a real mix of 200s (served), 403s (blocked), and 301s (redirected), so the publisher's defenses were firing and worked some of the time, but the blocks didn't matter because each request came from a different IP. A 403 on one address did nothing to stop the next from a different IP, and the scraper kept rotating until enough 200s landed to assemble the content.

1.2.4 Example 4: Scraper Triggers Google Ad Bots

The two Google Ads Bot requests are worth taking a look at here. They appear to have fired because the scraper rendered the page like a real browser, loading its ads, which is what set them off. The scrape looks human enough to trigger the ad stack; it could be taken as evidence that the scraper's disguise is convincing and successful. This presents an ad fraud concern for advertisers who are paying to see their ads viewed by humans, and not bots. (This is a topic we mentioned previously in our Q2 2025 Bot report, Section 1).

TollBit has made the Scraper Audit tool available on the TollBit platform to help websites test and understand their vulnerability to IP threats from major web scrapers. If you are interested in seeing how your site is being scraped by these third parties, please contact team@tollbit.com.

1.3 The Inefficiency and Cost of Web Scraping

This arms race between scrapers and website operators is not only inefficient for website owners, who are increasingly spending on cybersecurity to detect bots that are only getting harder to detect, but it is also deeply inefficient for AI developers. As mentioned in our prior bot report, we have found advanced scraping services charging up to $22.50 for 1,000 pages. Given the volume of data needed to serve consumer AI applications, the data acquisition costs for the most popular chatbots and specialized agents are likely to run into the tens of millions of dollars a year. And this is before considering the legal fees needed to fight lawsuits.

We have also observed that scrapers take 16 seconds on average to access publisher sites, and some of them may fail entirely in securing access to the page requested, whereas accessing content via a sanctioned route (i.e. via a platform like TollBit) typically takes less than 0.7 seconds.

There's a better way. Instead of spending huge sums on technical workarounds to bypass IP controls, AI developers can simply pay for the content they use. This reduces latency and increases reliability. By relying on licensing rather than scraping, journalism is funded — not the scrapers — and the media ecosystem AI depends on is sustained.

1.4 The Limits of Cybersecurity and the Need for Regulation

We've seen publishers increasingly take matters into their own hands to combat the surge in AI bots, increasingly investing in cybersecurity to actively block this traffic from their sites or even take action by filing lawsuits. As mentioned in our Q3 & Q4 2025 report, in late 2025, both Reddit and Google turned to the courts over unauthorized scraping. Google alleged that scrapers had circumvented SearchGuard, the in-house protection system it built at a cost of millions. If a company with Google's resources and engineering talent cannot keep determined scrapers out, an individual publisher has no realistic chance of doing so alone.

Cybersecurity is important, but it is not a foolproof solution. These tools are costly to run, and many of our publishers, even those with the most sophisticated defenses in place, have seen scrapers and AI bots get through anyway.

This brings us to an important limitation of the data in the subsequent sections of this report: we rely on AI companies self-identifying through their user agents. Developers sometimes use third-party vendors to scrape on their behalf; they may masquerade as other user agents, rotate their IP addresses, use headless browsers that mimic humans, use residential proxies, etc. This activity is difficult, and often impossible, for sites to trace back to a specific AI company. The true level of scraping, and of unpermitted access to European content, is, therefore, likely greater than this report shows.

As AI traffic becomes harder to distinguish from human traffic, it is important that regulators take a position that, at the bare minimum, AI bots should not be allowed to mimic humans on the Internet. Ideally, they would be required to self-identify and state their intended use, which is what New York's Stealth Crawler Prohibition Act would require. That bill, passed by the New York State Legislature in June 2026 and awaiting the Governor's signature, would require crawlers accessing a news site to disclose their identity and purpose through an accurate user-agent string before they access the content.

Self-identification enables a site owner to decide how to respond: allow, block, or charge for access via infrastructure controls like the ones TollBit provides. It would help ensure that the undue burden of paying to serve, detect, and block unwanted bots no longer falls on the websites themselves.