robots.txt and AI bots: don't close the door by accident
Blocking crawlers such as GPTBot, ClaudeBot and PerplexityBot can remove your site from AI answers. Which bots exist and how to allow them?
· 1 min read · Kieosa Editorial
Many sites unknowingly block AI crawlers because of default theme or security plugin settings. If a bot can't read your site, it's harder for the assistant to give current information about you.
Important AI crawlers
GPTBot,OAI-SearchBot,ChatGPT-User— OpenAIClaudeBot,Claude-SearchBot,Claude-User— AnthropicPerplexityBot,Perplexity-User— PerplexityGoogle-Extended— permission for use in Google's AI productsApplebot-Extended,Meta-ExternalAgent,CCBot
Example allow rules
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: *
Disallow: /adminThings to watch
- Block the admin area, cart and personal pages; keep public content open.
- Check that your CDN or firewall (e.g. bot protection) isn't blocking these bots separately.
- Allowing them may also mean consenting to your content being used for model training; make this decision consciously.
You can check which bots your site is open to with the free AI visibility test.
KIEOSA
Wondering whether your site is ready for AI?
Scan 30 signals for free and see your score right away.