What you are actually deciding
Allowing an AI crawler means your content can be used to build answers, and your business can be named and cited in them. Blocking it means your content is not used, which also means you are far less likely to be named.
For a publisher whose product is the content, blocking can make commercial sense: the content is the thing being monetised and giving it away to an answer engine cannibalises the visit. For a local service business, the content is not the product. The phone call is. Being cited is the goal, not the loss.
So for most local businesses, allow
If you want to be recommended, you have to be readable. Blocking the crawler that builds the answers you want to appear in is one of the few self-inflicted visibility problems in this field.
The important part is that it should be a decision. Most robots.txt files were never decided at all — they were inherited from a theme, a template, a developer, or a plugin default from years ago. We have seen sites actively blocking crawlers whose owners had no idea and would never have chosen it.
When blocking is defensible
If your content is the product, original research, proprietary data, paid material, or you have a specific legal or contractual reason. Those are real cases and they are not the typical local operator.
There is also a middle position: allow the crawlers that cite and drive referral traffic, and be more restrictive about the ones that do not. That requires knowing which is which, and it changes.
What we will not tell you
That blocking AI crawlers protects your rankings, or that allowing them harms your SEO. Neither claim is supported, and both get made confidently.
What we will do is check what your file currently says, tell you what it is doing, and let you make the call knowing what it costs either way.
Read in order, or jump
Twelve chapters, in the order the work has to happen. The first four are the spine; the platform chapters at the end include two surfaces we deliberately do not score, and say why.
01 How AI search picks local businesses 02 Eligibility: can AI even see your site? 03 Robots.txt: block AI crawlers or allow them? 04 Measuring AI visibility honestly 05 Entity foundations for local businesses 06 Reviews as AI input 07 Citation surfaces: where AI engines look 08 Getting ChatGPT to recommend your business 09 Showing up in Perplexity 10 Showing up in Google AI Overviews 11 Showing up in Gemini 12 Copilot, and why we don't sample itSee where you actually stand
A free Visibility Check runs the measurement described in this guide on your own business.
Get my free Visibility Check