Cloudflare has made its "block AI training" setting available to all customers, according to its blog. According to PANews, the setting allows sites to remain indexed by search engines while refusing to let content be used for AI model training, addressing the trade-off created when search and training rely on the same crawler. Cloudflare also introduced a "responsible" crawler operator designation. To qualify, operators must provide an opt-out for training and AI summaries, disclose URL-level indexing status, and promise that opting out of training will not affect search rankings. Apple, Google, Microsoft, Amazon, Anthropic, Meta, and OpenAI received the designation. Cloudflare said about 17% of sites have enabled some form of training-blocking mechanism, while fewer than 1% block search crawlers.
