AI Crawlers Are Not All the Same

Subscribe to JorgeTechBits newsletter

Disclaimer: I create this content entirely on my own time, and the views expressed here are mine alone (not my employer’s). Because I love leveraging new tech, I use AI tools like Gemini, ChatGPT, Claude, Perplexity and others as a “digital team” to help research and polish these articles so I can share the best possible insights with you!

For much of the web’s history, automated access was relatively easy to understand.

Search engines crawled websites, built indexes, and helped people find the original pages. There were exceptions, of course, but that was the basic bargain: a website made information available, a search engine made it discoverable, and some of those searches resulted in people visiting the site.

AI is making that bargain much less straightforward.

A crawler indexing one of my articles so someone can find it through search is doing something very different from a crawler collecting thousands of articles to train or fine-tune an AI model.

And both are different from an AI agent visiting a website in real time because someone asked it to find information, compare options, fill out a form, or perform a task.

Yet, for a while, website owners were essentially faced with a very blunt choice:

Let the bots in, or block them.

That choice no longer matches what is actually happening on the web.

Cloudflare recently changed its AI crawler controls, and I think the interesting part isn’t really the new dashboard settings.

It’s the distinction Cloudflare is starting to make between search, AI training, and AI agents.

Those are three very different uses of a website.

From “AI Bots” to What the Bot Is Actually Doing

Cloudflare now organizes AI-related automated traffic around three broad behaviors:

Search — crawling content to build an index and provide search results later.
Training — collecting content to train or fine-tune AI models.
Agent — a system visiting a website in real time on behalf of a person, such as a chat-fetch bot or browser-using agent.

At first glance, this may look like a fairly small technical distinction.

I don’t think it is.

Instead of asking:

Do I allow AI bots?

The better question becomes:

What am I allowing an automated system to do with my content?

Search can help people discover my site.

Training can incorporate material from my site into a model, potentially without anyone visiting the original page when that knowledge is later used.

An agent might access the site in real time to answer someone’s question or actually do something for them.

They may all generate automated requests to the same website.

But they are not the same use.

The Mixed-Use Crawler Problem

This gets particularly complicated with what Cloudflare calls mixed-use crawlers.

Companies such as Google, Microsoft, and Apple operate traditional search crawlers while also offering AI products.

That creates an interesting problem for website owners.

If I block a crawler because I don’t want my articles used for AI training, could I also make those articles harder to find through normal search?

For a blogger, publisher, small business, or really anyone who depends on organic search, that’s not a particularly attractive tradeoff.

I may not want my articles used to train a model.

But I certainly don’t want my site disappearing from search.

Cloudflare’s new Disallow AI Training option is designed to create some separation between those activities.

The idea is that accountable mixed-use crawlers can continue indexing the site for search while receiving a signal that the content should not be used for AI training.

The implementations aren’t identical. Google and Apple already provide mechanisms intended to distinguish search indexing from certain AI uses, while Microsoft is moving in a similar direction.

But the larger idea matters more than the individual crawler implementation:

Allowing search should not automatically mean allowing AI training.

That seems like a fairly reasonable distinction.

Disallow and Block Are Not the Same Thing

There is an important technical detail here.

A traditional robots.txt file isn’t a security barrier.

It publishes instructions telling crawlers what a website owner permits or doesn’t permit. Well-behaved crawlers follow those instructions.

A crawler that chooses to ignore them still can.

That’s why Cloudflare’s approach is interesting. It combines two ideas.

The first is publishing the website owner’s preference for cooperating crawlers.

The second is enforcing that preference at Cloudflare’s network edge when the traffic can be identified and blocked.

So Disallow AI Training isn’t simply another name for Block.

For mixed-use crawlers, the goal is to preserve legitimate search indexing while communicating that the content shouldn’t be used for training.

For training-specific crawlers where that search tradeoff doesn’t exist, Cloudflare can block access.

As AI use of the web expands, I think website owners are going to need both sides of this equation: a way to clearly state what automated systems are allowed to do and a practical way to enforce those choices where possible.

Then There Are the Agents

This may be the part I find most interesting.

Historically, websites have really been designed for two primary audiences:

People and search engines.

We now have a third audience emerging:

AI agents acting for people.

Imagine asking an AI assistant:

Find three restaurants nearby with outdoor seating, compare their menus, and tell me which ones have live music tonight.

The agent might visit several websites, read event calendars, compare listings, and return an answer.

You may never visit any of those websites yourself.

Or imagine an agent reading support documentation, checking product availability, filling out a scheduling form, comparing products, or interacting with a customer portal.

That’s not traditional search.

And it’s certainly not AI training.

It’s delegated interaction.

A person has given software permission to interact with websites on their behalf.

That creates an entirely different set of questions.

Reading a public article is one thing.

Completing a transaction is another.

Accessing a customer account, making an appointment, submitting a form, or purchasing something brings authorization, identity, security, fraud prevention, and accountability into the picture.

Crawler controls are only the beginning.

The agentic web is eventually going to need better ways for websites and agents to understand who is acting, who authorized the action, what the agent is allowed to do, and where the boundaries are.

We Need a Better Vocabulary

Another thing I like about Cloudflare’s approach is that it starts giving us better language for talking about how content is used.

Cloudflare’s Content Signals framework distinguishes between:

search — building search indexes and providing search results.
ai-input — using content as input to an AI system when answering a request.
ai-train — using content to train or fine-tune a model.

That middle category is particularly interesting.

Suppose I publish an article explaining how something works.

There is a meaningful difference between an AI system reading that article today because someone asked a question and an AI company collecting the article so it can become part of a future model.

One is immediate retrieval.

The other is model development.

And there are more distinctions coming.

Can an AI system quote the material?

Can it summarize it?

Can it store it?

Should it provide attribution?

Should it link back?

Can an agent act on information from the site?

These questions were much less important when the primary automated visitor to a website was a search crawler building an index.

They become much more important when AI systems can consume information, synthesize it, and act on it.

Why Website Owners Should Pay Attention

Cloudflare is replacing its older broad Block AI Bots approach with more granular controls for Search, Training, and Agents.

For many Cloudflare customers, the migration will happen automatically based on their previous settings.

So this isn’t necessarily something that requires an urgent configuration change.

But I do think it’s worth opening the dashboard and looking at the new controls.

Not because everyone should block AI.

I don’t think that’s the right conclusion.

The better question is what role you want these systems to play on your site.

For example, a retailer may welcome AI agents that make its products easier to discover or buy.

A publisher whose business depends on page views may feel very differently about an AI system providing complete answers without sending the reader to the original article.

A technical blogger might be perfectly comfortable with an AI assistant retrieving an article to answer someone’s question while still preferring that the entire site not be collected for model training.

Different sites have different economics and different reasons for publishing.

Their AI policies should probably be different too.

The Larger Shift

That’s why I think this Cloudflare change is more interesting than it initially appears.

The old web bargain was imperfect, but understandable:

Search engines crawled sites.
Search engines built indexes.
People searched.
Some of those people visited the original sites.

AI complicates that exchange.

A model can consume content during training.

An AI assistant can retrieve information from multiple sources and synthesize an answer without the person necessarily visiting those sources.

And an AI agent can go one step further — visiting websites and actually doing things on behalf of the user.

This doesn’t mean AI access is inherently harmful.

It means different kinds of access create different kinds of value — and different costs — for the people who create and operate websites.

That’s why lumping everything together under “AI bots” doesn’t make much sense anymore.

Cloudflare hasn’t solved the larger problem.

robots.txt and emerging content signals still depend partly on cooperation. Standards are evolving. Crawler identities can be difficult to verify. AI-generated answers introduce separate questions about attribution and traffic. And we’re only beginning to figure out the rules for agents interacting with websites on our behalf.

But we’re starting to develop a more useful middle ground between:

Let every automated system use everything.

and

Block everything.

For my own sites, the principle is fairly straightforward.

I want people to be able to find my work.

I want useful AI tools to be able to help people discover and use it.

But that doesn’t automatically mean every automated system should have unrestricted permission to collect it for training, reproduce it in answers, or interact with my sites however it chooses.

Those are different activities.

And as the web becomes increasingly agentic, I think understanding those differences is going to matter a lot more.