01
Why doesn't blocking a page keep it out of Google?
Because Google can list an address it has never read: if other sites link to a blocked page, its URL can still appear in results. For a page you want hidden, such as a thank-you page, Google's advice is different: "To keep a web page out of Google, block indexing with noindex or password-protect the page."
Don't combine the two. Google says that if a page "is blocked by a robots.txt file or the crawler can't access the page, the crawler will never see the noindex rule, and the page can still appear in search results." So pick one job per page: noindex and leave it crawlable, or block crawling and accept that the address may still be listed.
02
What does a contractor's robots.txt need?
Usually very little, and having no file at all is fine: Google says that means it "can crawl all URLs on your site." If you have one, it "lives at the root of your site" and "must be named robots.txt." A contractor's site often needs only a rule letting every crawler in and a sitemap line, adapted from Google's example (each line is its own line in the file):
| Line in the file | What it tells crawlers |
|---|
| User-agent: * | The rules below apply to every crawler |
| Allow: / | Every page may be crawled |
| Sitemap: https://www.example.com/sitemap.xml | Where your sitemap is |
The costly mistake is one line. "Disallow: /" under "User-agent: *" tells every crawler that follows the file to stay off the whole site. Web designers sometimes use it to hide a site while it's being built, so check robots.txt the day a new site goes live. If you fix a mistake, Search Console's robots.txt report lets you request a recrawl; otherwise Google says it generally caches the file "for up to 24 hours." Our guide to website migration SEO lists robots.txt among the launch checks.
03
How do you write robots.txt rules for AI crawlers?
Give each bot its own group: a User-agent line with the bot's name, then its rules. Google says its crawlers follow only the group with "the most specific user agent" that matches them, so for Google a named group replaces the "*" rules rather than adding to them. OpenAI says of its bots, "Each setting is independent of the others."
Here's a file that lets every crawler in but says no to OpenAI's model training and to Google's Gemini training and grounding:
| Group | Lines in the file | What it tells crawlers |
|---|
| 1 | User-agent: * then Allow: / | Every crawler may crawl every page |
| 2 | User-agent: GPTBot then Disallow: / | OpenAI's training crawler stays off the site |
| 3 | User-agent: Google-Extended then Disallow: / | Don't use the site for Gemini training and grounding |
| Last line | Sitemap: https://www.example.com/sitemap.xml | Where your sitemap is |
Each line goes on its own line in the file, with a blank line between groups.
OpenAI says disallowing GPTBot "indicates a site's content should not be used in training generative AI foundation models." Google-Extended isn't a separate crawler; Google calls it a token "used in a control capacity" for Gemini training and grounding, with no effect on inclusion or ranking in Google Search. Search bots such as OAI-SearchBot stay allowed under the "*" group. Which bots to allow, bot by bot, is covered on our AI crawler page, and our guide on how to show up in ChatGPT answers covers what helps once they're in.
Related terms: AI crawler · Canonical tag · llms.txt · All glossary terms
Where to go next: The technical SEO checklist puts robots.txt alongside the other crawl and index checks worth running once a year.