ACTION 1 OF 16

How to allow AI crawlers on your site: robots.txt, CDN, and a 15-minute check

SHORT ANSWER

Allow AI search crawlers in robots.txt: OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, Bingbot, and Googlebot. Make sure your CDN or firewall doesn’t block them separately. Then request a key page as the bot. A 200 response means AI can see it.

  • Time: 15 min
  • Owner: you (check) · developer (CDN)
  • Impact: ≈ +1–4% Coverage ¹

Kyrylo Poltavets, Founder of Dabudai · LinkedIn
Published · Updated

Why do AI crawlers need access to your site?

ChatGPT, Perplexity, and Claude find answers through their own crawlers. If a crawler is blocked in robots.txt or at the CDN, your page won’t get cited, no matter how good it is. This usually happens by accident: a rule meant to protect your content from bots also locks out AI crawlers, so they never see anything to cite.

How to open your site to AI crawlers, step by step

  1. Open yoursite.com/robots.txt in your browser.
  2. Find the rules for AI search and assistant crawlers: OAI-SearchBot and ChatGPT-User (ChatGPT), PerplexityBot and Perplexity-User (Perplexity), Claude-SearchBot and Claude-User (Claude), Bingbot (Copilot), and Googlebot (AI Overviews and AI Mode). [OpenAI] [Perplexity] [Anthropic]
  3. Make sure none of them sits under “Disallow: /”. Also check that the catch-all “User-agent: *” rule doesn’t block your services, pricing, or blog folders.
  4. Ask whoever manages your CDN or firewall (Cloudflare, Akamai, and so on) whether AI bot blocking is turned on. It can cut off crawlers even when robots.txt allows them. [Cloudflare]
  5. Request a key page as the bot, using the command in the template below. A 200 means the page is reachable. A 403 or a challenge page means something is blocking it.

Template: robots.txt rules and an access check

# Allow AI search and assistant crawlers
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

# Optional: block model training only.
# This doesn't affect ChatGPT search.
User-agent: GPTBot
Disallow: /

# Check in your terminal (expect HTTP 200):
curl -I -A "OAI-SearchBot" https://yoursite.com/pricing

How do you know AI crawler access is working?

Your 10 most important pages return 200 when requested as the bot. robots.txt doesn’t block any search or assistant crawler. And your CDN has no AI bot blocking turned on for your site.

FREE AUTOMATED CHECK

Don’t want to check every bot by hand?

Enter your site, and the audit will check access for every AI bot in this guide: OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, Bingbot, and Googlebot. It also runs through the other technical GEO and SEO checks. The report lands in your inbox in a few minutes.

What’s the most common mistake with AI crawler access?

Treating GPTBot and OAI-SearchBot as the same bot. GPTBot collects training data. OAI-SearchBot powers ChatGPT search. Many sites block both to “protect their content” and then disappear from ChatGPT search results.

How can you tell AI has started reading your site?

The first changes show up in 2–4 weeks. Your pages start appearing among the sources in AI answers, and Coverage grows on platforms where you didn’t show up before. Compare each model separately: ChatGPT, Google AI Overviews, and AI Mode update at different speeds.

If you use Dabudai, open Overview → AI Provider Coverage and track Coverage for each platform from the day you opened access to the bots.

Dabudai Overview: Coverage, Share of Voice, position, and Recommendation Rate by AI platform

No account yet? Your first brand data shows up within 20 minutes of signing up. The first 7 days are free.

Common questions about AI crawler access

Can I block GPTBot and still show up in ChatGPT search?

Yes. GPTBot only collects training data. ChatGPT search runs on OAI-SearchBot and ChatGPT-User, so those are the ones to allow. [OpenAI]

Do I need an llms.txt file?

No. Google says its AI search features don’t require any special files, and there’s no public evidence that AI assistants rank sites by llms.txt. [Google]

How long until I see results?

Crawlers come back anywhere from a few days to a few weeks later. The first changes in Coverage usually show up in 2–4 weeks.

What Dabudai customers say

★★★★★

We can finally see how AI recommends us

“We had no visibility into how AI tools recommend our business or how they describe it. Now we can see both, and some of the insights have been incredibly useful.”

Daniel Nowak

· CMO, medical tourism company

GO DEEPER

For more on AI bot access, see Lesson 4.3 in the Learning Center inside the Dabudai platform. It includes examples and a check on your own data.

The Learning Center is open during your 7-day free trial, no credit card required.

Sources

[OpenAI] OpenAI: Overview of OpenAI crawlers ↗

[Perplexity] Perplexity: Perplexity crawlers ↗

[Anthropic] Anthropic: Claude’s web crawlers ↗

[Cloudflare] Cloudflare: Block AI bots ↗

[Google] Google: Guide to optimizing for AI features in Search ↗

¹ Estimated impact comes from Dabudai’s analysis of 350,000 AI answers across 2,300 websites, October 2026. Your results depend on your niche, competitors, and starting point.