Should you block AI crawlers in robots.txt?
For most local service businesses the answer is allow. The trade-off is real in both directions, so this chapter sets out what each choice costs, when blocking is genuinely defensible, and the two claims about robots.txt we will not make. Whichever way you go, it should be a decision rather than a default.
What are you actually deciding?
Allowing an AI crawler means your content can be used to build answers, and your business can be named and cited in them. Blocking it means your content is not used, which also means you are far less likely to be named.
Which way that cuts depends on what you actually sell.
| Business type | Where the money is | What that means for the decision |
|---|---|---|
| A publisher whose product is the content | The content is the thing being monetised and giving it away to an answer engine cannibalises the visit. | Blocking can make commercial sense. |
| A local service business | The content is not the product. The phone call is. | Being cited is the goal, not the loss. |
Why is allow the right default for a local service business?
If you want to be recommended, you have to be readable. Blocking the crawler that builds the answers you want to appear in is one of the few self-inflicted visibility problems in this field.
The important part is that it should be a decision. Most robots.txt files were never decided at all — they were inherited from a theme, a template, a developer, or a plugin default from years ago. We have seen sites actively blocking crawlers whose owners had no idea and would never have chosen it.
When is blocking AI crawlers defensible?
There are cases where blocking is the right call. If your content is the product, original research, proprietary data, paid material, or you have a specific legal or contractual reason. Those are real cases and they are not the typical local operator.
There is also a middle position: allow the crawlers that cite and drive referral traffic, and be more restrictive about the ones that do not. That requires knowing which is which, and it changes.
What will we not tell you about robots.txt?
There are two claims about robots.txt and AI crawlers that this chapter will not make. That blocking AI crawlers protects your rankings, or that allowing them harms your SEO. Neither claim is supported, and both get made confidently.
What we will do is check what your file currently says, tell you what it is doing, and let you make the call knowing what it costs either way.
Read in order, or jump
The order here is the order of the work: twelve chapters, the first four structural, the last of them covering two platforms we decline to score and why we decline.
01 How AI search picks local businesses 02 Eligibility: can AI even see your site? 03 Robots.txt: block AI crawlers or allow them? 04 Measuring AI visibility honestly 05 Entity foundations for local businesses 06 Reviews as AI input 07 Citation surfaces: where AI engines look 08 Getting ChatGPT to recommend your business 09 Showing up in Perplexity 10 Showing up in Google AI Overviews 11 Showing up in Gemini 12 Copilot, and why we don't sample itSee where you actually stand
A free Visibility Check runs the measurement described in this guide on your own business.
Get my free Visibility CheckRankings are never guaranteed. Anything we could not trace to a primary source is absent from this page, not estimated. The audit runs the same six layers described on the pricing page, and Share of Answer is scored quarterly.