**Managing AI Bot Traffic: The New Era of Control and Transparency**
The digital landscape is evolving rapidly, and with it, the ways websites interact with automated traffic. Today, businesses face diverse priorities: some aim to maximize discovery by search and AI agents, while others prioritize strict security to prevent their content from being used for model training. Managing bot traffic has become increasingly complex, especially when different policies overlap or contradict one another.
Cloudflare has long provided tools to help website owners manage these challenges—from robots.txt directives to advanced Bot Management solutions. However, we recognized the need for a more unified, transparent, and manageable approach. That’s why we’re introducing **Bot Preference Sync**, a feature designed to streamline how you control AI bot activity on your site.
—
### New Questions Facing the Internet
For years, the central question for website owners was: *Is my content being used to train AI models without my permission?* This concern remains critical. But as AI technology advances, new questions have emerged:
– How can I improve my content’s visibility in AI-powered search results?
– What portion of my traffic comes from AI bots compared to real users?
– Which content drives referrals, and what is its actual value?
The answers to these questions vary significantly depending on your business model. An e-commerce platform, for example, wants its products to appear when users ask chatbots for recommendations. Conversely, a publisher relying on ad revenue may want their content indexed for search visibility but excluded from AI training—while still verifying that this preference is respected.
There is no one-size-fits-all solution. That’s why control and transparency must be at the core of any modern bot management strategy.
—
### The Case for Transparency
On July 1, 2026, we highlighted the risks posed by mixed-use crawlers—bots that simultaneously handle search, agent interactions, and training under a single user agent. These bots make it difficult for site owners to enforce distinct preferences for each use case.
Our stance on transparency remains unchanged: we want to reward bots that clearly identify themselves and disclose how they use crawled data. For bot **Verification**, operators of mixed-use crawlers must now meet additional requirements if they wish to avoid being blocked when “Disallow Training” is set. These include:
– Respecting robots.txt directives that prohibit training
– Offering site owners an opt-out from AI summaries
– Providing URL-level visibility into which pages are available for training
– Demonstrating that disallowing training does not harm traditional search visibility
Bots that fail to meet these transparency standards will not receive favorable treatment—they will be blocked when training is disallowed. Transparency is becoming the price of admission.
—
### Introducing Bot Preference Sync
Bot Preference Sync is a new feature that automatically synchronizes your AI bot settings with your robots.txt file. Available to all customers—from Free to Enterprise tiers—it ensures that your stated preferences for Search, Agent, and Training traffic are consistently published and enforced.
Key features include:
– **Automatic updates**: Your robots.txt is dynamically updated based on your Cloudflare dashboard settings
– **Consistency**: What you configure is what you publish, eliminating conflicts between directives and enforcement rules
– **Flexibility**: You can enable or disable the sync at any time
Instead of managing a separate static file, Cloudflare generates and maintains your robots.txt, keeping policy and enforcement aligned.
#### Configuring Your Preferences
You can set policies for each bot category:
– **Search**: Allow, block ads, or block entirely
– **Agent**: Allow, block ads, or block entirely
– **Training**: Allow or disallow
For example, selecting “Disallow Training” adds a `Disallow: /` rule for verified training bots in your robots.txt. Cooperating mixed-use crawlers that meet transparency requirements can still access your content for indexing—without using it for model training.
—
### Default Behaviors for Different Customers
To better serve diverse needs, we’ve established new default behaviors:
– **New customers**: No blocks or disallows are applied by default. You decide which bot categories to restrict.
– **Ad-supported publishers**: Option to enable “I monetize from pages with ads,” which sets Training to Disallow by default—helping you stay visible in search while keeping content out of AI training.
– **Legacy users**: Those using the managed robots.txt feature will be prompted to transition to Bot Preference Sync.
—
### What’s Next?
Bot Preference Sync will roll out to all customers across every plan in the coming week. Watch for dashboard prompts and changelog updates.
This is one step in a broader effort to balance discoverability, security, and transparency. As AI technologies continue to reshape the web, we remain committed to giving site owners the tools and clarity they need to protect their content while engaging with emerging opportunities.
—
## FAQ
**What is Bot Preference Sync?**
Bot Preference Sync is a feature that automatically syncs your AI bot settings from the Cloudflare dashboard to your robots.txt file. It ensures your preferences for Search, Agent, and Training traffic are consistently published and enforced.
**Who can use Bot Preference Sync?**
It is available to all Cloudflare customers, from the Free tier to Enterprise.
**Does Bot Preference Sync replace my existing robots.txt?**
No. It prepends dynamic directives to your existing robots.txt file, preserving any manual rules you’ve already set.
**Can I disable Bot Preference Sync?**
Yes. You can turn it on or off at any time in the Cloudflare dashboard.
**What happens if a bot doesn’t meet transparency requirements?**
Bots that don’t provide transparency will be blocked when training is disallowed, even if they claim to support search or agent functions.
**Will my search visibility be affected?**
If you allow Search and the bot is verified and respects your policies, your search visibility will remain unaffected.
—
## Conclusion
Bot Preference Sync represents a new way to manage AI bot traffic with greater precision, transparency, and ease. By aligning your stated preferences with enforced policies, we help you protect your content while staying visible in an AI-driven ecosystem.
As always, your strategy should guide your controls. With Bot Preference Sync, you now have a more streamlined and reliable way to reflect your priorities in one powerful, automated solution.
