The Cloudflare check: is your site invisible to AI crawlers?
A setting you probably did not choose, on a layer most owners forget exists, that can remove you from AI answers entirely.
Short answer: check your CDN, not just robots.txt. A CDN or firewall rule is a wall, and it operates before robots.txt is ever read, so a clean robots.txt on its own proves nothing about crawler access. Open your bot-management settings, fetch your own homepage without JavaScript, and read your server logs.
- A CDN rule blocks before robots.txt is ever read.
- Fetch your own page without JavaScript and see what a crawler sees.
- A blocked crawler cannot retrieve you, so it cannot cite you.
- It is usually a toggle, which makes it the cheapest fix available.
Why does a CDN block not show up in robots.txt?
A CDN or firewall block never appears in robots.txt, because the two sit on different layers. Robots.txt is a request. A CDN or firewall rule is a wall, and it operates before robots.txt is ever read. Owners check robots.txt, see nothing alarming, and conclude crawler access is fine.
| Layer | What it is | Order of operations |
|---|---|---|
| robots.txt | A request. | Read only after a request has already passed the CDN or firewall layer. |
| A CDN or firewall rule | A wall. | It operates before robots.txt is ever read. |
What we read from that, and this part is our judgement rather than anything Cloudflare or any other vendor states: a clean robots.txt is not evidence that AI crawlers can reach you, so the CDN layer is the one worth opening first.
Cloudflare and similar services added controls for AI crawler traffic, and some accounts have them switched on by default or enabled during a security review by someone who read the traffic as scraping.
How do you check for an AI-crawler block in five minutes?
Three checks, in order. Open your CDN or security dashboard and look for any bot-management or AI-crawler setting. Read what it is currently blocking rather than what you remember configuring. Then fetch your own homepage with a plain request that does not execute JavaScript and see what comes back.
If the response is a challenge page, a block page, or an empty shell, that is what a crawler sees too.
Finally check server logs for the crawler user agents you expect. Absence over weeks is the strongest signal available.
Why does an AI-crawler block matter more than it used to?
Eligibility comes first. A blocked crawler cannot retrieve you, and a system that cannot retrieve you cannot cite you. This sits at the eligibility layer, before content, entity work or anything else, and a failure here quietly voids the rest.
It is also the cheapest fix on the list. A toggle, in most cases.
Should you allow AI crawlers, or block them?
Allowing crawlers means your content is used to build answers, which is the point if you want to be named in them. For a publisher selling access to content, blocking can be right. For a local service business selling jobs, it rarely is.
Either way it should be a decision. We find sites blocking crawlers whose owners had no idea and would not have chosen it.
“We find sites blocking crawlers whose owners had no idea and would not have chosen it.”
The protocol behind every check we run is published before we run it: read the full method.
Rankings are never guaranteed. Anything we could not trace to a primary source is absent from this page, not estimated. The audit runs the same six layers described on the pricing page, and Share of Answer is scored quarterly.
Youssef measures visibility for established local businesses across Google Maps, organic search and AI answers, quarterly, against a protocol published in advance. Numbers that cannot be checked stay out of the report.
More about Youssef