Why it matters
ChatGPT, Perplexity, and Claude find answers through their own crawlers. If a crawler is blocked in robots.txt or at the CDN level, your page won’t get cited, no matter how good it is. This usually happens by accident. A rule meant to “protect content from AI” blocks search too, not just model training.
Step by step
Open yoursite.com/robots.txt in your browser.
Find the rules for AI search and assistant crawlers: OAI-SearchBot and ChatGPT-User (ChatGPT), PerplexityBot and Perplexity-User (Perplexity), and Claude-SearchBot and Claude-User (Claude). Also look for Bingbot (Copilot) and Googlebot (AI Overviews and AI Mode).
Make sure none of them sits under “Disallow: /”. Also check that the general “User-agent: *” rule doesn’t block your services, pricing, or blog folders.
Ask whoever manages your CDN or firewall (Cloudflare, Akamai, etc.) whether AI bot blocking is turned on. It can cut crawlers off even when robots.txt allows them.
Request a key page as the bot using the command in the template below. 200 means the page is accessible. 403 or a challenge page means something is blocking it.
Template: robots.txt rules and a quick check
# Allow AI search and assistant crawlers
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
# Optional: block model training only.
# This doesn't affect ChatGPT search.
User-agent: GPTBot
Disallow: /
# Check in the terminal (expect HTTP 200):
curl -I -A "OAI-SearchBot" https://yoursite.com/pricingHow to know it’s done
Your 10 most important pages return 200 when requested as the bot. No search or assistant crawler is blocked in robots.txt, and AI bot blocking is off in your CDN.
Common mistake
Treating GPTBot and OAI-SearchBot as the same bot. GPTBot collects training data. OAI-SearchBot powers ChatGPT search. Many sites block both to “protect their content” and then disappear from ChatGPT search results.
How to tell AI has started reading your site
You’ll see the first changes in 2–4 weeks. Your pages start showing up among the sources of AI answers, and Coverage grows on platforms where you weren’t present before. Compare each model separately. ChatGPT, Google AI Overviews, and AI Mode update at different speeds.
If you use Dabudai, open Overview → AI Provider Coverage and track Coverage on each platform from the day you opened access to the bots.
app.dabud.ai · Overview
No account yet? Your first brand data shows up within 20 minutes of signing up. The first 7 days are free.
Questions
Can I block GPTBot and still show up in ChatGPT search?
Yes. GPTBot only collects training data. ChatGPT search runs on OAI-SearchBot and ChatGPT-User, so those are the ones to allow.
Do I need llms.txt?
No. Google says you don’t need special files for AI search features. There’s also no public evidence that AI assistants rank sites based on llms.txt.
How long until I see results?
Crawlers come back anywhere from a few days to a few weeks. The first changes in Coverage usually show up within 2–4 weeks.
From people already doing this
★★★★★
We can finally see how AI recommends us
«Before, we had no idea how AI tools recommended our business or how they described it. Now we can see both, and some of the insights turned out to be incredibly useful.»
Daniel Nowak
· CMO, medical tourism company
GO DEEPER
Learn more about AI bot access in Lesson 4.3 of the Dabudai Learning Center, with examples and a check on your own data.
Sources
¹ Estimated impact of each action and the niche tables comes from Dabudai’s analysis of 350,000 AI answers across 2,300 websites (ChatGPT, ChatGPT Search, Google AI Overviews, Google AI Mode), October 2026. These are average ranges: your results depend on your niche, competitors, and starting point.