One crawler can index you and train on you.
Refusing the training half used to mean risking the search half.
On September 15, Cloudflare split its crawler controls by behaviour and added a setting called Disallow AI Training, which publishes a no training preference in robots.txt while leaving search crawling untouched.
Apple, Google and Microsoft either honour that preference today or have committed to honour it inside a stated timeframe, and Cloudflare tracks what each operator actually does on Radar.
The same release redefined the blunter Block setting so that it now covers mixed use crawlers, which means it reaches Applebot, Bingbot and Googlebot, search included.
Cloudflare's own numbers make the split concrete: fewer than 1% of its sites choose to block search bots, while 17% enable some mechanism to block training.
Crawler control is no longer a switch you flip to keep bots out; it is a policy with a default, and the default belongs to whoever creates the domain.
What Disallow AI Training actually does
Cloudflare now classifies bots by behaviour: Search, Training and Agent, and applies your choice at the domain level.
Disallow AI Training is the only one of those settings that keeps a mixed use crawler visiting for search while refusing it for training.
It works by publishing the preference in robots.txt rather than by dropping the connection, and that difference is the whole argument.
The mechanism underneath is Bot Preference Sync, which keeps robots.txt aligned with what you select in the dashboard and replaces the older Managed Robots.txt setting.
That older approach had limits worth remembering: a file can state a preference, but it cannot identify who is crawling, classify why, or stop a crawler that ignores it.
A network can do all three, and that is essentially what is on offer here.
Defaults moved as well, since new domains now block Training and Agent behaviour on pages that display ads while Search stays allowed.
The reasoning is revenue, because a summary can answer the question that would have brought the visit which paid for the page.
The useful question is not whether to allow AI crawlers, but which of the available settings matches the sentence you would say out loud about your content.
What people are getting wrong
The most common mistake is treating robots.txt as enforcement, when it is a published preference that a network then has to defend.
The second is reading a dashboard as proof, because the setting records your intent while the crawler's behaviour decides the outcome.
The third is assuming the definition held still, since Block used to be the safe way to keep AI crawlers out and it now reaches search crawlers most sites want.
Verification is cheap and unglamorous: sample your logs for the user agents you care about, and fetch your own robots.txt after saving, instead of trusting the panel.
If you run a domain with any search traffic, this reaches you whether or not you are on Cloudflare, because the preference lands in a file every crawler reads.
If you are hosted elsewhere, the same decision exists in a different shape, and the question is identical: which behaviours does this setting cover?
Three things to change this week
- Audit the setting, not the intent. A zone still carrying the legacy aggressive block now loses search crawling along with training, which makes it a traffic incident rather than a policy tweak. Move those zones to Disallow AI Training if the intent was always to refuse training and keep search.
- Give Microsoft a second lever. Robots.txt support for a no training preference at Microsoft is targeted for early 2027, so the new setting does not yet tell Bing anything. Use the NOARCHIVE meta tag or the URL blocking tool for that operator, and confirm the effect in your logs.
- Write the decision into the domain template. New domains are offered preset configurations, and those presets are stricter for sites that earn money from advertising, because a summary can replace the visit that paid for the page. Put the choice in your provisioning code with the ad revenue assumption behind it, so it survives the next account migration.
Agent behaviour is the remaining gap, since there is no Disallow style option for it yet and chat fetch bots and browser use agents can only be blocked outright.
The ai-prefs work at the IETF is where that is expected to change, which is also the reason to prefer standards that enough crawlers actually read.
Anyone who has watched a scraper eat a paid API quota will recognise the instinct to block first, but that reflex now carries a visible cost when the same crawler also indexes you.
The old assumption and the new one
The old assumption was that refusing training was free: turn the bots off and keep the rankings.
The new reality is that only one configuration keeps both, so the choice has to be deliberate, documented and revisited as the standards mature.
Cloudflare also introduced a designation it calls Accountable, which an operator earns by offering a training opt out, an AI summary opt out, page level visibility into what was made available for training, and an assurance that opting out will not affect search results.
Apple, Google and Microsoft qualify, with the parts still in development attached to stated deadlines rather than shipped evidence.
Read that carefully, because the label is issued by the platform that publishes the preference and it describes commitments as much as it describes running code.
Summaries are the next argument, and a site wide yes or no is a poor fit, since how much of an article appears changes whether a reader ever arrives.
How training data shapes model behaviour is its own subject, and a short AI fundamentals competence check is a reasonable next step if that is the part you care about.
If the machines fetching your pages are the interesting part, the tooling behind them is worth understanding.
Reusing content for training without a deal is also a business problem, and we looked at it from the API side in how metered API traffic becomes training data.
When crawler access becomes a policy rather than a switch, who in your organisation signs the robots.txt your domain publishes?