AI Bots Have Become a Major Contributor to Website Traffic
The viral virtual assistant OpenClaw—previously referred to as Moltbot, and even earlier as Clawdbot—represents a significant shift that could transform the dynamics of the internet. Rather than being predominantly a human space, the web may soon be overtaken by autonomous AI bots.
A recent report examining bot interactions on the internet, along with data shared with WIRED by internet infrastructure provider Akamai, indicates that AI bots are already responsible for a substantial portion of web traffic. The report also highlights a sophisticated arms race as bots utilize clever strategies to circumvent website defenses designed to block them.
“In the future, bots will constitute the majority of internet traffic,” remarks Toshit Pangrahi, cofounder and CEO of TollBit, a firm monitoring web-scraping activities and author of the new report. “This isn’t solely a copyright issue; a new kind of visitor is emerging on the internet.”
Many major websites strive to restrict what content bots can access for AI training. (WIRED’s parent company, Condé Nast, alongside other publishers, is currently pursuing legal action against several AI companies for alleged copyright violations related to AI training.)
However, another form of AI-related web scraping is also gaining traction. Numerous chatbots and AI tools can now pull real-time information from the web, enhancing their outputs with current product prices, movie schedules, or summaries of the latest news.
Data from Akamai reveals that bot traffic related to training has been consistently rising since last July. Additionally, global activity from bots gathering web content for AI agents is on the rise.
“AI is reshaping the web as we know it,” states Robert Blumofe, Akamai’s CTO, in conversation with WIRED. “The resulting arms race will influence the future appearance, usability, and functionality of the web, as well as the fundamental aspects of conducting business.”
TollBit anticipates that by the fourth quarter of 2025, an average of one in every 50 visits to its clients’ websites will originate from an AI scraping bot. In the first quarter of 2025, this statistic stood at one in every 200. The company observes that in the fourth quarter, over 13 percent of bot requests bypassed the robots.txt file, which some websites utilize to indicate which pages bots should stay away from. TollBit reports a staggering 400 percent increase in the proportion of AI bots ignoring robots.txt from the second to the fourth quarter of last year.
Furthermore, TollBit noted a 336 percent rise in the number of websites attempting to block AI bots over the past year. Pangrahi explains that scraping methods are evolving as sites seek to control how bots access their content. Some bots camouflage their traffic to appear as if it originates from a regular web browser or send requests designed to replicate human website interactions. The study conducted by TollBit indicates that the behavior of certain AI agents has become nearly indistinguishable from that of human web traffic.
TollBit offers tools that allow website owners to charge AI scrapers for accessing their content. Other companies, such as Cloudflare, provide similar options. “Anyone relying on human web traffic—particularly publishers, but essentially everyone—will feel the impact,” warns Pangrahi. “A more efficient means of enabling machine-to-machine, programmatic exchanges of value is essential.”
Rajat Sharma
https://in.linkedin.com/in/rajat-mediaHelping D2C Brands Scale with AI-Powered Marketing & Automation 🚀 | $15M+ in Client Revenue | Meta Ads Expert | D2C Performance Marketing Consultant