What is an AI crawler?

OpenAI and Google document the split. OpenAI says OAI-SearchBot "is for search," GPTBot crawls content that "may be used in training," and ChatGPT-User visits pages when a person asks ChatGPT something. At Google, Googlebot covers Search and its AI features, while Google-Extended is a separate robots.txt token for Gemini training and grounding.

Google-Extended has no user agent of its own. It's a name you use in robots.txt, and Google says it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search." Other AI companies run their own bots; their names show up in your server logs and in tools such as Cloudflare's.

It depends on which one. OpenAI says sites that opt out of OAI-SearchBot "will not be shown in ChatGPT search answers," though they can still appear as navigational links. Blocking GPTBot signals that your content shouldn't be used for training. At Google, robots.txt rules for Googlebot are the control for Search, including AI Overviews and AI Mode.

To stay in Google Search but out of AI Overviews and AI Mode, Google offers a separate Search generative AI control in Search Console; sites are included by default. Also, robots.txt is a request, not a lock. Cloudflare's documentation says "robots.txt compliance is voluntary," and OpenAI says that for ChatGPT-User, which acts on a person's request, "robots.txt rules may not apply."

Through settings that block bots at the network level, whatever your robots.txt says. Cloudflare's AI Crawl Control, available on all plans, lets site owners allow or block individual AI crawlers. Its AI bot policies group bots as Search, Agent or Training, and Cloudflare says that from September 15, 2026, new domains block Training and Agent bots on pages with ads by default.

Search stays allowed under those defaults. But Cloudflare also says settings that block AI training will block "mixed-purpose crawlers that combine Search and Training." Other hosts and security plugins have their own bot settings. That's why Google lists "ensuring that crawling is allowed in robots.txt, and by any CDN or hosting infrastructure," and OpenAI asks owners to confirm their host or CDN lets its search bot through.

That nothing is blocking the search crawlers you want. Open yoursite.com/robots.txt and look for "Disallow" lines under Googlebot, OAI-SearchBot or "User-agent: *". Then check your Cloudflare, hosting or security plugin settings for AI bot blocking. Whether to allow training crawlers such as GPTBot is a separate business choice.

Most home-service companies want to be found in every search tool homeowners use, so a blanket "block AI bots" switch can cost more than it protects. Changes take time to register; OpenAI says about 24 hours after a robots.txt update. Once the crawlers can get in, how to show up in ChatGPT answers covers what comes next.

Related terms: llms.txt · AI citation · AI Overviews · AI Mode · Query fan-out · All glossary terms

Where to go next: How to show up in ChatGPT answers and how to show up in Google AI Overviews pick up from here. All our AI search guides are in the AI search hub.

• OpenAI, Overview of OpenAI crawlers (read October 29, 2026)
• OpenAI Help Center, Searching the web with ChatGPT (read October 29, 2026)
• Google Search Central, Google's common crawlers (last updated October 15, 2026; read October 29, 2026)
• Google Search Central, AI features and your website (last updated December 10, 2025; read October 29, 2026)
• Google Search Console Help, Search generative AI control in Search Console (read October 29, 2026)
• Cloudflare Docs, AI Crawl Control (last updated August 14, 2026; read October 29, 2026)
• Cloudflare Docs, Block AI Bots (last updated July 1, 2026; read October 29, 2026)
• Cloudflare Docs, robots.txt setting (last updated October 22, 2026; read October 29, 2026)

Want to see what Google's AI and ChatGPT say about your business today? Get a free AI Visibility Review.