The Cloudflare check: is your site invisible to AI crawlers?
A setting you probably did not choose, on a layer most owners forget exists, that can remove you from AI answers entirely.
- A CDN rule blocks before robots.txt is ever read.
- Fetch your own page without JavaScript and see what a crawler sees.
- A blocked crawler cannot retrieve you, so it cannot cite you.
- It is usually a toggle, which makes it the cheapest fix available.
The block that does not show in robots.txt
Owners check robots.txt, see nothing alarming, and conclude crawler access is fine. Robots.txt is a request. A CDN or firewall rule is a wall, and it operates before robots.txt is ever read.
Cloudflare and similar services added controls for AI crawler traffic, and some accounts have them switched on by default or enabled during a security review by someone who read the traffic as scraping.
How to check in five minutes
Open your CDN or security dashboard and look for any bot-management or AI-crawler setting. Read what it is currently blocking rather than what you remember configuring.
Then fetch your own homepage with a plain request that does not execute JavaScript and see what comes back. If the response is a challenge page, a block page, or an empty shell, that is what a crawler sees too.
Finally check server logs for the crawler user agents you expect. Absence over weeks is the strongest signal available.
Why it matters more than it used to
A blocked crawler cannot retrieve you, and a system that cannot retrieve you cannot cite you. This sits at the eligibility layer, before content, entity work or anything else, and a failure here quietly voids the rest.
It is also the cheapest fix on the list. A toggle, in most cases.
The decision underneath
Allowing crawlers means your content is used to build answers, which is the point if you want to be named in them. For a publisher selling access to content, blocking can be right. For a local service business selling jobs, it rarely is.
Either way it should be a decision. We find sites blocking crawlers whose owners had no idea and would not have chosen it.
“We find sites blocking crawlers whose owners had no idea and would not have chosen it.”
Youssef runs visibility work for established local businesses: Google Maps, organic search and AI answers, measured monthly against a published protocol. Where a number cannot be checked, it does not go in the report.
More about Youssef