Crawlability is whether search engine and AI crawlers can reach the pages on your site and read their content. A page that can’t be crawled can’t be indexed, and a page that isn’t indexed can’t be a source in any search result or AI answer.
There used to be one gatekeeper: Googlebot. Today, crawlers from OpenAI, Anthropic, Perplexity and other AI companies come to your site alongside it. Each arrives under its own name, gets its own line in robots.txt and meets its own rule on your server. Crawlability now covers more than “Can Google see my page?”
Why crawlability matters for your brand
The first door of AI visibility is “reach.” If a model can’t reach your site, your content can’t make it into an answer, however good it is. That door is usually closed by a default setting, not a decision.
The biggest example: on July 1, 2025, Cloudflare announced it would block AI crawlers by default on the sites it serves. Unless site owners allow them, those bots can’t get in (MIT Technology Review, July 2025). If your site sits behind Cloudflare, your AI door may have closed without anyone deciding to close it.
How crawlability works
- Permission. A crawler checks your robots.txt first. If there’s a block for its name (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and so on), it turns back.
- Access. Even with permission, your server, firewall or CDN can block the request and return a 403 or 429 (too many requests) error.
- Reading. Once the page opens, the content has to be in the HTML. Many AI crawlers don’t run JavaScript, so pages built in the browser look empty to them.
- Discovery. Internal links and your sitemap help crawlers find new and updated pages. A page nothing links to often goes unfound.
In an analysis published by Vercel and MERJ in December 2024, none of the major AI crawlers, including those from ChatGPT, Claude and Perplexity, ran JavaScript. Google’s Gemini rendered pages fully because it uses Googlebot’s infrastructure (Vercel, December 2024). Important content should be rendered on the server.
The SEO and GEO connection
Crawlability is the foundation of technical SEO and the first door of GEO. The same rules apply: a clean robots.txt, a fast and error-free server, server-rendered content and good internal linking. What changed is how many visitors are knocking. A site that’s open to Google can still be closed to AI crawlers.
One distinction matters: some user agents, such as Google-Extended, control AI training only. Blocking them doesn’t affect crawling for Google Search. Opting out of training and blocking a live search crawler are different decisions, and you should make them separately.
What we’ve seen in the field
About 3 in 4 of the more than 100 websites we audited through September 2026 didn’t explicitly allow AI crawlers (The Three Doors of AI Visibility). The cases we see most:
- No AI crawler named anywhere in robots.txt. Nothing was decided. It was just left alone.
- A CDN or firewall blocking AI bots even though robots.txt allows them.
- Content loaded with JavaScript. Google sees it. The AI crawler sees an empty page.
It happened on our own site. In September 2026, our server logs showed that some GPTBot and ClaudeBot requests were being turned away with 429 errors at the hosting layer, and we raised it with our host. Your robots.txt can open the door while your server keeps it shut. We wrote more about this in Blocking AI Agents Is the New SEO Mistake.
A practical crawlability playbook
- Put the decision in writing. Name the AI crawlers you allow in robots.txt. Decide separately for training bots and search bots.
- Check your infrastructure. Review the bot rules in your CDN, firewall and hosting. If you use Cloudflare, check the AI crawler setting explicitly.
- Read your server logs. Confirm from the logs that AI bots actually arrive, and which response code they get.
- Serve content in HTML. Main content, headings and links should be on the page without JavaScript.
- Make discovery easy. Keep a current sitemap, use sensible internal links and leave no orphan pages.
- Test regularly. Recheck crawl permissions after every major site, plugin or infrastructure change.
Related terms
Frequently asked questions
Should I allow AI crawlers?
If you want to show up as a source in AI answers, you need to allow at least the search and user-request bots (for example, OAI-SearchBot, ChatGPT-User, Claude-SearchBot and PerplexityBot). You can decide separately about bots that collect training data.
Will blocking Google-Extended hurt my Google rankings?
No. Google-Extended controls whether your content is used to train Google’s AI models. It doesn’t affect crawling or ranking in Google Search.
Do AI crawlers run JavaScript?
Most don’t. In Vercel and MERJ’s December 2024 analysis, the ChatGPT, Claude and Perplexity crawlers didn’t run JavaScript. Serve important content in the HTML, rendered on the server.
How can I tell whether AI bots are reaching my site?
Search your server logs for their user-agent names and check the response codes. Anything other than 200 (such as 403, 429 or 503) means the bot showed up but couldn’t get in.
