Is Your Website Invisible to ChatGPT?
Search used to mean one thing: rank on Google. Now, more and more people are asking ChatGPT, Claude, or Perplexity a question instead of typing it into a search box — "find me a roofer near me," "what's a good chiropractor in [city]," "where should I eat tonight." If an AI assistant can't read your site, it can't recommend you. That's Answer Engine Optimization (AEO), and it's quietly become as important as traditional SEO.
Here's the uncomfortable part: your site can be doing everything right — fast, well-structured, technically sound — and still be invisible to AI assistants, because of one file most people never look at: robots.txt.
What we found auditing our own platform
We build every page to load instantly (real numbers: full page content delivered in a quarter of a second, no JavaScript required to see it) and we'd already written a robots.txt file explicitly welcoming the crawlers that matter for AI visibility — Anthropic's ClaudeBot, OpenAI's GPTBot, Google's AI crawler, Perplexity's crawler.
But when we checked what was actually being served, we found a contradiction: our hosting provider's own bot-management feature was silently overriding our policy, hard-blocking the exact crawlers we'd explicitly tried to allow. Anthropic's ClaudeBot had already tried to reach our site 13 times — and been turned away every time, invisibly, with nothing in our own code telling it to.
This is a genuinely easy mistake to make, because it doesn't show up as an error. Your site loads fine for humans. Search Console looks normal. The only way to catch it is to actually check what's being served to AI crawlers specifically, not just what your own code says it's serving.
It took two fixes, not one
Our first move was to go through Cloudflare's crawler list one by one and turn off "Block Crawler" for every bot that matters for AI visibility — not just the obvious three (ChatGPT, Claude, Perplexity), but a much longer list most people don't know exists: Google's AI crawler, Apple's, Meta's (both the training crawler and the one that fires when someone asks Meta AI to fetch a specific page live), DuckDuckGo's AI assistant, Mistral's, and OpenAI's and Anthropic's own real-time search crawlers, distinct from their general-purpose ones.
We did that. Every toggle, checked and corrected. We re-verified the live robots.txt file to confirm — and the exact same conflicting block was still there. Unblocking a crawler and having it stop being rejected at the edge is a real, necessary fix. It is not the same thing as the robots.txt file itself no longer contradicting your own policy — those turned out to be two completely independent settings behind two different toggles, and fixing one doesn't touch the other.
The actual second setting was a master toggle labeled "Managed robots.txt," sitting one screen up from the crawler list, with a description that — in hindsight — said exactly what it was doing: automatically writing content into our robots.txt file on our behalf. Turning that off was what finally resolved it.
The lesson here isn't really about Cloudflare specifically. It's that modern hosting/CDN platforms increasingly have several independent layers of bot policy stacked on top of each other, and "I checked one setting" is not the same as "I verified the actual file being served." Always re-fetch the live file after every change and confirm it matches your intent — don't trust that a dashboard toggle did what its neighboring toggle's description implied.
Why this matters more for local businesses
If you're a national SaaS brand, being cited by an AI assistant is nice-to-have. If you're a roofer, a chiropractor, or a restaurant, it's how people are increasingly finding businesses "near me" — and it depends on something even more foundational than crawler access: whether your site tells AI systems the basic facts (your address, phone number, hours) in a structured way they can actually use. That's a separate, related piece we're building into our platform directly, so every business site built on it gets this by default — not as something you have to remember to configure.
The AI crawler checklist
If you want to check your own site, here's exactly what we did, in order:
- Confirm your own
robots.txt(the one your own code/CMS generates) explicitly saysAllow: /for the crawlers that power AI answers — see the table below for the full list. - If you're on Cloudflare: go to AI Crawl Control → Overview and turn off "Managed robots.txt." Its whole job is to auto-write content into your file — usually not the content you actually want.
- Go to the crawler list (check "Show inactive crawlers" if some are missing) and turn off "Block Crawler" for every AI-answer-engine bot, one at a time. Don't assume a bulk "allow all" setting covers it — check each one.
- Re-fetch your live
robots.txt— not the one in your codebase, the one actually being served (curl https://yoursite.com/robots.txt) — and confirm there's no duplicate or contradictory rule for any crawler you care about. - Check "Unsuccessful requests" per crawler in your dashboard. A high count on a bot you want to allow is a red flag something is still rejecting it, even after the policy looks right.
- Repeat this whole check any time you change hosting/CDN providers, redesign your site, or launch a new domain — none of this carries over automatically.
The crawler reference list
The "big three" everyone thinks of aren't the whole picture. Here's the fuller list we actually went through:
| Crawler | Owner | Verdict |
|---|---|---|
| GPTBot, OAI-SearchBot, ChatGPT-User | OpenAI | Allow |
| ClaudeBot, Claude-User, Claude-SearchBot | Anthropic | Allow |
| Google-Extended, Google-CloudVertexBot | Allow | |
| Googlebot | Always allow — regular search, not AI-specific | |
| PerplexityBot, Perplexity-User | Perplexity | Allow |
| Meta-ExternalAgent, Meta-ExternalFetcher, FacebookBot | Meta | Allow |
| Applebot-Extended | Apple | Allow (Apple Intelligence/Siri) |
| DuckAssistBot | DuckDuckGo | Allow |
| MistralAI-User | Mistral | Allow |
| Bytespider, TikTok Spider | ByteDance | Block — aggressive, no reciprocal benefit |
| CCBot | Common Crawl | Block — broad training dataset, not an answer engine |
| Amazonbot, Anchor Browser, Manus Bot, Novellum, ProRataInc, PetalBot, Terracotta Bot, Timpibot | Various | Block — no clear AEO benefit for most sites |
The takeaway
AEO isn't a future concern — it's already shaping who gets found today. And it's not enough to write the right policy; you have to verify it's actually what's being served, at every layer, because the layers between your code and the internet (CDNs, bot-management features, default settings) can silently override your intent without ever throwing an error.