AI Crawlers: Who Scrapes Your Site for AI

A claim keeps making the rounds on tech blogs and X: Meta scrapes the web for AI training 31 times harder than OpenAI and eats 70% of all training traffic. It's a clean number, so people repost it, usually crediting DataFast analytics. One problem — the source they point to doesn't contain those figures. DataFast's crawler directory is a taxonomy of 61 bots across four categories, with not a single per-company percentage in sight.

AI crawlers are the bots that walk your site and file your content into training datasets and AI search indexes. Over the past year AI crawlers have become a real slice of every site's traffic: per Fastly, automated bots already make up 37% of requests on their network, and nearly 80% of AI crawler activity is data collection for model training.

At Gless we build RAG systems and agents that fetch external sources for a living, and we run our own sites that these bots hit every day. So we read reports like this from both sides at once. Let's go through the numbers: who actually scrapes hardest, where the "31x" came from, and what a site owner should do about it.

Who really scrapes hardest — Meta, OpenAI, or Anthropic?

Short answer: it depends on whose network you measure. There's no single "global traffic" picture — just data from individual CDNs and security vendors, and they disagree.

Fastly's Threat Insights report for Q2 2025 (published August 25, sampled across 6.5 trillion monthly requests and 130,000 applications) breaks AI-crawler traffic down like this:

Three companies account for 95% of the volume, and Meta alone outweighs Google and OpenAI combined. That's where "Meta scrapes hardest" comes from — it rests on real data, just not on DataFast.

Now a different network. Cloudflare Radar, same period (July 2025), puts someone other than Meta on top among AI-only bots:

By Cloudflare's count OpenAI scrapes more than Meta, not 31 times less. A year earlier, in July 2024, Meta sat at 2.4%; over twelve months it climbed to 17.7%, while Bytespider collapsed from 37.3% to 5.8%. Meta's trajectory is genuinely the most aggressive — but "70% of all traffic" shows up in no independent network.

Why the spread? Fastly and Cloudflare have different customer bases, different bot-verification methods, and different aggregation windows. And bots are trivial to fake: Fastly estimates roughly 6% of traffic claiming to be an AI crawler is spoofed under someone else's user-agent. So any single figure here is a snapshot of one network, not a law of physics.

Where "31x" and "70%" came from — and why to treat them carefully

Direct answer: the exact source of those numbers can't be found. The datafa.st/crawlers page people cite is a descriptive catalog — 61 bots grouped into "AI answers" (16 bots), "Search indexes" (14), "Training crawlers" (18), and "Other" (13). No "31.3x," no "70.4%," no per-company splits, no sample size, no measurement date.

This is how most viral SEO stats are born: take a real signal ("Meta is currently the biggest training crawler, per Fastly"), round it into a punchy number, strip the source and the window, and let it travel. For us it's a working rule — any crawler stat is worth checking against whose network and which month it describes. Missing that, the number is decorative.

The practical takeaway doesn't change: Meta became one of the most active AI crawlers over the past year, and that alone is reason to understand how these AI crawlers move through your site. Just without the magical "31 times" precision.

Three types of AI bots you can't lump together

The one thing to carry out of this whole topic: "AI bot" isn't one thing. There are three types, and they deserve different treatment, because some take your content silently while others send you people.

The difference matters. GPTBot files your text into model weights with no attribution — no link, no visit. OAI-SearchBot and ChatGPT-User bring a user in from an AI answer with a clickable source. Meta runs two separate bots too: Meta-ExternalAgent trains Llama, while Meta-ExternalFetcher pulls a specific page when someone asks Meta AI something, sometimes with a citation.

One more subtlety — the opt-out tokens Google-Extended and Applebot-Extended. These aren't crawlers: they make zero HTTP requests and exist only as robots.txt directives that say "don't use my content to train Gemini / Apple Intelligence." Blocking them "to cut load" is pointless — their load is zero.

If you want the mechanics deeper, training versus retrieval sits at the core of what an AI agent is and how it works with external data.

They crawl a lot — and send almost nothing back

Direct answer: how much a bot scrapes and how much traffic it returns are different leagues, and the gap is enormous. Cloudflare tracks crawl-to-refer — pages downloaded per visit the platform sends back. For July 2025:

Google takes five pages and sends a visit. Anthropic takes nearly forty thousand pages for every visit — near pure one-way traffic. That's the economics the "31x" headline hides: it's not only who crawls most, it's who returns anything at all. We covered the same lopsidedness in money and tokens in our breakdown of LLM token economics — same logic, applied to traffic instead.

Which gives a site owner a simple test: training crawlers are pure content donation with no feedback loop; search and user bots are a potential source of visits from AI answers. Treating all AI crawlers the same way is a mistake.

What this means for your site

Beyond the attribution philosophy, AI crawlers create three concrete effects that are already visible.

Load and junk requests. AI crawlers are still noticeably sloppier than Googlebot. Per Vercel, ChatGPT's crawler hits a 404 on 34.8% of requests and Claude on 34.2%, versus 8.2% for Googlebot. Almost one in three training-bot requests goes nowhere — but still loads your server. None of the major training crawlers execute JavaScript; they take the HTML as-is.

Stealth crawlers. Not everyone plays fair. In August 2025 Cloudflare showed Perplexity running undeclared bots that disguise themselves as ordinary Chrome on macOS and rotate IPs outside the official ranges to slip past robots.txt and blocks — an estimated 3–6 million requests a day on top of declared traffic. So robots.txt honestly stops the honest.

The GEO/AEO dilemma. Block everything and you protect content from training, but you drop out of ChatGPT and Perplexity answers — a growing channel. Allow everything and you hand proprietary content to datasets with no attribution. The right answer is almost never all-or-nothing.

How to control AI crawlers in practice

Start with what almost no one does: open robots.txt and split AI crawlers by type. Per Cloudflare, only ~37% of top sites even have a robots.txt, and GPTBot is blocked on just 7.8% of them. Basic control over AI crawlers isn't set up almost anywhere.

The working strategy is block training, keep retrieval:

# Don't feed content into training
User-agent: GPTBot
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

# But stay visible in AI search
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

This pulls your text out of OpenAI and Meta training while keeping a shot at appearing in ChatGPT answers with a link back. Both GPTBot and OAI-SearchBot claim to respect robots.txt and are controlled independently.

Then, as your setup allows:

• Cloudflare. Even the free plan has an "AI Scrapers and Crawlers" toggle that blocks known bots in one click. Cloudflare turns it on by default for new domains since 2024; after the feature launched, Bytespider traffic dropped 71%. AI Crawl Control lets you pick allow / block / charge-per-crawl per bot (pay-per-crawl is still in beta).
• Don't lean on llms.txt. The format gets talked about, but real consumption is nil: adoption sits near 10% of domains, and as Google's John Mueller noted in June 2025, no AI system currently uses it. Ship it if you like; don't count on it.
• Verify from logs, not faith. robots.txt is a voluntary agreement, and a user-agent is faked in a second. Real protection comes from server logs and CDN/WAF rules where you can see how a bot actually behaves.

If you want to work out which AI crawlers to allow given your content strategy, and tie that to visibility in AI search, that's exactly the kind of thing we handle in our AI implementation services. Get in touch if you'd like us to look at your logs and build a policy around your goals.

The viral "31x" turned out to be a number with no source. But the question behind it is real: AI crawlers are already walking your site every day, and deciding who gets in is better done on purpose than by default.