--- title: "Cloudflare adds separate controls for search and AI training" type: "News" locale: "en" url: "https://longbridge.com/en/news/299156802.md" description: "Cloudflare introduced separate controls for search indexing, AI training, and agents, allowing customers to reject model training without losing search visibility. The company launched an 'Accountable' designation for crawler operators meeting transparency requirements, awarded to Apple, Google, and Microsoft. This replaces the previous single 'Block AI Bots' switch with three distinct categories, addressing mixed-use crawlers that previously forced a choice between search presence and AI training opt-outs." datetime: "2026-09-16T08:29:05.000Z" locales: - [zh-CN](https://longbridge.com/zh-CN/news/299156802.md) - [en](https://longbridge.com/en/news/299156802.md) - [zh-HK](https://longbridge.com/zh-HK/news/299156802.md) generator: "portal-rs" --- # Cloudflare adds separate controls for search and AI training Cloudflare has introduced separate controls for search indexing, AI training and AI agents, allowing customers to reject model training without removing their sites from traditional search results. The company also launched an “Accountable” designation for crawler operators that meet, or commit to meet, its transparency requirements. Apple, Google and Microsoft received the designation, Cloudflare said on September 15. The settings are available through Cloudflare’s network and replace a single “Block AI Bots” switch. Website owners can change their selections, while recommended defaults vary according to whether a site carries advertising. The change addresses mixed-use crawlers that collect material for search and AI training under the same bot identity. A site that blocked the crawler to prevent training could previously also lose its search visibility. Cloudflare’s framework allows designated operators to keep search access when a customer declines training. ## Three controls replace one switch Cloudflare now classifies crawler activity under three uses: search, training and agents. Search covers indexing intended to help people find content. Training includes collection for model training or fine-tuning. Agent activity covers automated systems acting in real time for a user. For new sites that display ads, Cloudflare recommends allowing search, disallowing AI training and blocking agents on pages carrying ads. Its recommended setting for other sites allows all three uses. Customers can override those choices. Cloudflare’s documentation gives customers three actions for each policy: allow the crawler, block it on all pages, or block it only on pages with advertising. The options apply to verified bots in the selected category and to additional unverified traffic that Cloudflare places in the same class. The company is also replacing its Managed Robots.txt feature with Bot Preference Sync. The new feature lets customers set preferences once and applies them across crawlers supported by Cloudflare. The controls combine preference signals with Cloudflare’s ability to block crawler traffic at the edge. The changes follow a policy announced in July. At that time, TechCrunch reported that Cloudflare planned to change defaults for new domains on September 15 and apply more restrictive treatment to crawlers used for multiple purposes. ## Cloudflare sets criteria for mixed-use crawlers Cloudflare said mixed-use crawlers account for 36.6 percent of verified crawler traffic on its network. It also said fewer than 1 percent of website owners block search crawlers, while 17 percent restrict AI training. The measurements describe Cloudflare’s network and should not be read as estimates for the entire web. To receive the Accountable designation, an operator must provide a way to opt out of AI training, permit an opt-out from AI-generated search summaries, offer URL-level visibility into how content is used, and confirm publicly that declining training will not affect traditional search rankings. Cloudflare said Apple, Google and Microsoft either satisfy those requirements or have provided timelines for doing so. Other operators may qualify when they separate search and training crawlers or provide comparable controls. Mixed-use crawlers are blocked when a Cloudflare customer disallows training unless the operator qualifies under the company’s framework. Google, for example, offers the Google-Extended token so sites can restrict some model-training uses without blocking Google Search. Cloudflare said other AI companies operate separate crawlers for search and training, allowing their training bots to be blocked without removing search access. Chief executive Matthew Prince said the approach was intended to preserve search while giving publishers and businesses more control over the use of their work. ## The controls remain platform-specific The settings apply to websites using Cloudflare and do not create a universal rule for crawler behavior. Signals placed in robots.txt or related standards still depend on crawler compliance unless traffic is blocked by Cloudflare. The Accountable label is Cloudflare’s own classification, not an independent certification. Cloudflare is participating in work at the Internet Engineering Task Force on “ai-prefs,” a draft specification intended to give websites a portable way to express AI-use preferences. The standard remains under development. The company also aims to let website owners control how much of their material appears in AI search summaries by early 2027. That capability is a stated goal and is not part of the current release. > What happens when machines become the web’s main audience? ### Related Stocks - [NET.US](https://longbridge.com/en/quote/NET.US.md) - [MSFT.US](https://longbridge.com/en/quote/MSFT.US.md) - [MSFX.US](https://longbridge.com/en/quote/MSFX.US.md) - [MSFU.US](https://longbridge.com/en/quote/MSFU.US.md) - [AAPL.US](https://longbridge.com/en/quote/AAPL.US.md) - [GOOGL.US](https://longbridge.com/en/quote/GOOGL.US.md) - [AAPD.US](https://longbridge.com/en/quote/AAPD.US.md) - [AAPB.US](https://longbridge.com/en/quote/AAPB.US.md) - [MSFD.US](https://longbridge.com/en/quote/MSFD.US.md) - [MSFO.US](https://longbridge.com/en/quote/MSFO.US.md) - [AAPU.US](https://longbridge.com/en/quote/AAPU.US.md) - [AAPW.US](https://longbridge.com/en/quote/AAPW.US.md) - [APLY.US](https://longbridge.com/en/quote/APLY.US.md) - [AAPX.US](https://longbridge.com/en/quote/AAPX.US.md) - [AAPY.US](https://longbridge.com/en/quote/AAPY.US.md) - [MAGX.US](https://longbridge.com/en/quote/MAGX.US.md) - [MSFL.US](https://longbridge.com/en/quote/MSFL.US.md) - [MSFW.US](https://longbridge.com/en/quote/MSFW.US.md) - [MSFY.US](https://longbridge.com/en/quote/MSFY.US.md) - [GGLL.US](https://longbridge.com/en/quote/GGLL.US.md) - [GGLS.US](https://longbridge.com/en/quote/GGLS.US.md) - [GOOW.US](https://longbridge.com/en/quote/GOOW.US.md) - [GOOY.US](https://longbridge.com/en/quote/GOOY.US.md) - [GOU.US](https://longbridge.com/en/quote/GOU.US.md) - [GOOX.US](https://longbridge.com/en/quote/GOOX.US.md) - [GOOP.US](https://longbridge.com/en/quote/GOOP.US.md) - [GOOG.US](https://longbridge.com/en/quote/GOOG.US.md) ## Related News & Research - [Cloudflare Helps End the Search-or-AI-Training Tradeoff | NET Stock News](https://longbridge.com/en/news/299055070.md) - [Google just completed its $1.5 billion-plus talent deal for AI startup Mechanize](https://longbridge.com/en/news/298761757.md) - [Microsoft AI Releases Draft Code Of Conduct For MAI Models](https://longbridge.com/en/news/298935159.md) - [Cloudflare Sees Agentic AI Opportunity Dwarfing Past Growth Drivers](https://longbridge.com/en/news/298813408.md) - [‘Not If, but When,’ Says Top Investor About Microsoft Stock](https://longbridge.com/en/news/299065198.md) --- > **Disclaimer: This article is for reference only and does not constitute any investment advice.**