Search used to mean one thing: ranking in Google. Now, more and more people ask ChatGPT, Claude, or Perplexity a question instead of typing it into a search box — "find me a roofer near me", "what's a good chiropractor in [city]", "where should I eat tonight". If an AI assistant can't read your site, it can't recommend you. That's Answer Engine Optimization (AEO), and it has quietly become as important as traditional SEO.
Here's the uncomfortable part: your site can be doing everything right — fast, well-structured, technically solid — and still be invisible to AI assistants, because of a file almost nobody looks at: robots.txt.
What we found auditing our own platform
We built every page to load instantly (real numbers: the full page content is delivered in a quarter of a second, no JavaScript required to see it) and we'd already written a robots.txt file that explicitly welcomed the crawlers that matter for AI visibility — Anthropic's ClaudeBot, OpenAI's GPTBot, Google's AI crawler, and Perplexity's crawler.
But when we checked what was actually being served, we found a contradiction: our own hosting provider's bot-management feature was silently overriding our policy, hard-blocking exactly the crawlers we'd explicitly tried to allow. Anthropic's ClaudeBot had already tried to reach our site 13 times — and been rejected every time, invisibly, with nothing in our own code telling us so.
This is a genuinely easy mistake to make, because it doesn't show up as an error. Your site loads fine for humans. Search Console looks normal. The only way to catch it is to actually check what's specifically being served to AI crawlers, not just what your own code says it's serving.
It took two fixes, not one
Our first step was to go through Cloudflare's crawler list one by one and turn off "Block crawler" for every bot that matters for AI visibility — not just the obvious three (ChatGPT, Claude, Perplexity), but a much longer list most people don't know exists: Google's AI crawler, Apple's, Meta's (both the training crawler and the one that fires when someone asks Meta AI to fetch a specific page live), DuckDuckGo's AI assistant, Mistral's, and OpenAI's and Anthropic's own real-time search crawlers, distinct from their general-purpose ones.
That's what we did. Every toggle, checked and fixed. We re-checked the live robots.txt file to confirm it — and the exact same conflicting block was still there. Unblocking a crawler and stopping it from being rejected at the network edge is a real, necessary fix. It isn't the same as the robots.txt file itself no longer contradicting your own policy — those turned out to be two completely independent settings behind two different switches, and fixing one doesn't touch the other.
The second real fix was a master switch labeled "Managed robots.txt", sitting one screen above the crawler list, with a description that — in hindsight — said exactly what it was doing: automatically writing content to our robots.txt file on our behalf. Turning that off is what finally resolved it.
The lesson here isn't really about Cloudflare specifically. It's that modern hosting/CDN platforms increasingly stack several independent layers of bot policy on top of each other, and "I checked a setting" isn't the same as "I verified the actual file being served." Always re-download the live file after every change and confirm it matches your intent — don't trust that a dashboard toggle did what the neighboring toggle's description implied.
Why this matters more for local businesses
If you're a national-reach SaaS brand, being cited by an AI assistant is a nice-to-have. If you're a roofer, a chiropractor, or a restaurant, it's increasingly how people find "near me" businesses — and it depends on something even more fundamental than crawler access: whether your site tells AI systems the basics (your address, phone number, hours) in a structured way they can actually use. That's a separate but related piece we're building directly into our platform, so every business site built on it gets this by default — not something you have to remember to configure.
The AI crawler checklist
If you want to check your own site, here's exactly what we did, in order:
- Confirm your own
robots.txt(the one your own code/CMS generates) explicitly saysAllow: /for the crawlers that power AI answers — see the table below for the full list. - If you're on Cloudflare: go to AI Crawl Control → Overview and turn off "Managed robots.txt". Its main job is to automatically write content into your file — usually not the content you actually want.
- Go to the crawler list (check "Show inactive crawlers" if some are missing) and turn off "Block crawler" for every AI answer-engine bot, one by one. Don't assume a blanket "allow all" setting covers it — check each one.
- Re-download your
robots.txtlive — not the one in your codebase, but the one actually being served (curl https://yoursite.com/robots.txt) — and confirm there's no duplicate or contradictory rule for any crawler you care about. - Check "Failed requests" per crawler in your dashboard. A high count on a bot you want to allow is a red flag that something is still rejecting it, even after the policy looks right.
- Repeat this entire check every time you change hosting/CDN providers, redesign your site, or launch a new domain — none of this carries over automatically.
The crawler reference list
The "big three" everyone thinks of aren't the whole picture. Here's the more complete list we actually went through:
| Crawler | Owner | Verdict |
|---|---|---|
| GPTBot, OAI-SearchBot, ChatGPT-User | OpenAI | Allow |
| ClaudeBot, Claude-User, Claude-SearchBot | Anthropic | Allow |
| Google-Extended, Google-CloudVertexBot | Allow | |
| Googlebot | Always allow — normal search, not AI-specific | |
| PerplexityBot, Perplexity-User | Perplexity | Allow |
| Meta-ExternalAgent, Meta-ExternalFetcher, FacebookBot | Meta | Allow |
| Applebot-Extended | Apple | Allow (Apple Intelligence/Siri) |
| DuckAssistBot | DuckDuckGo | Allow |
| MistralAI-User | Mistral | Allow |
| Bytespider, TikTok Spider | ByteDance | Block — aggressive, no reciprocal benefit |
| CCBot | Common Crawl | Block — broad training dataset, not an answer engine |
| Amazonbot, Anchor Browser, Manus Bot, Novellum, ProRataInc, PetalBot, Terracotta Bot, Timpibot | Various | Block — no clear AEO benefit for most sites |
The bottom line
AEO isn't a future concern — it's already determining who gets found today. And it's not enough to write the right policy; you have to verify it's actually what's being served, at every layer, because the layers between your code and the internet (CDN, bot-management features, defaults) can silently override your intent without ever throwing an error.