Your robots.txt says yes. Your CDN says 403.
The check that misses the problem
Most AI visibility audits check your robots.txt for AI crawler rules. If GPTBot is allowed, the audit records a pass and moves on.
That check is looking at a file that states an intention. It does not test whether the intention survives contact with your infrastructure.
What we found
In July 2026 we measured sixteen European tour operators. Two scored zero effective visibility while permitting every AI crawler in robots.txt. The causes were opposite, and neither was visible in the file.
Case 1: refused at the edge
One operator holds the joint highest technical score in the study: 68 out of 100. It publishes an llms.txt, ships structured data, and serves 2,193 words across 22 headings without requiring JavaScript. The content work is genuinely good.
Requesting the homepage as GPTBot, ClaudeBot or PerplexityBot returns HTTP 403. Akamai refuses the crawlers that robots.txt invites.
Effective score: zero. A competitor scoring 18 out of 100 on content quality is currently more readable to ChatGPT than they are.
Case 2: the page arrives empty
A second operator blocks nothing. GPTBot receives HTTP 200 and the full 115,801 bytes, byte for byte what a browser gets.
Of those bytes, 87,381 are JavaScript. The HTML contains no title tag, no meta description, no headings, no body text. The content is assembled in the browser, and AI crawlers do not run it.
We asked an AI assistant to read their About page and describe the company. It returned their Google Analytics consent script.
Why file-based audits miss both
A robots.txt is a request. Your CDN, WAF, bot-management rules and rendering pipeline are what actually decide what a crawler receives. Those four layers are usually owned by different teams than the one that wrote the robots.txt, often years apart.
The only reliable test is to request your own pages as each named crawler and read what comes back.
Test it yourself in one minute
curl -s -o /dev/null -w "%{http_code}\n" \
-A "GPTBot/1.2 (+https://openai.com/gptbot)" \
https://yoursite.com/
A 403 or 429 means you are blocked regardless of robots.txt.
If it returns 200, check what is inside it:
curl -s -A "GPTBot/1.2 (+https://openai.com/gptbot)" https://yoursite.com/ \
| grep -c "<title>"
Zero means an AI crawler cannot see your page title. Repeat for ClaudeBot and PerplexityBot, because bot-management rules often differ per agent.
What to do about each
A 403 at the CDN is a configuration change, not a project. Allow the search-and-cite crawlers by user-agent while keeping your scraper protections. Every crawler identifies itself, so this is selective.
A page that renders empty needs server-side rendering of the text that matters: title, description, headings, and the paragraphs describing what the company does. Not a rebuild. The pages that describe you are the ones that need it.
Neither is cloaking, and do not let anyone talk you into the shortcut of serving crawlers a special version of the page. Engines detect it and the penalty is removal.
