of Cloudflare customers block AI bots via their configuration. In the hotel sector, major groups block most AI bots, making themselves practically invisible on all major AI platforms. (Source: Cloudflare, Phocuswire 2025)
Who Are These AI Bots and What Do They Do?
Every major AI platform sends several types of bots across the web. Here are the most important ones for hoteliers:
OpenAI (ChatGPT)
- GPTBot — collects content to train ChatGPT/GPT-4 models. Blocking it means your information isn't part of the training data.
- OAI-SearchBot — builds the ChatGPT Search index. Blocking it means you don't appear in ChatGPT web searches.
- ChatGPT-User — triggered when a user asks ChatGPT to visit a specific page. Not an automatic crawler.
Anthropic (Claude)
- ClaudeBot — collects content to train Claude. Can be blocked independently of other Anthropic bots.
- Claude-SearchBot — builds Claude's search index. Blocking ClaudeBot doesn't prevent Claude-SearchBot from indexing you.
- Claude-User — triggered by a real user query. Its presence indicates Claude is actively recommending your content.
Google (Gemini)
- Google-Extended — collects content for Gemini Apps and Vertex AI training. Completely separate from Googlebot (classic SEO) — blocking it has zero impact on your regular Google rankings.
Perplexity
- PerplexityBot — builds Perplexity AI's search index. Perplexity cites its sources, giving you direct links back to your site.
Key point: Each bot is independently controllable. You can allow search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) while blocking training bots (GPTBot, ClaudeBot, Google-Extended). This is a strategic decision, not an all-or-nothing choice.
Arguments for Allowing AI Bots
1. Visibility in AI travel recommendations
Allowing search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) means your hotel can appear in conversational responses from these platforms. Otherwise, OTAs — which systematically allow these bots — get recommended in your place.
2. First-mover advantage
Most of your direct competitors — independent hotels in the same category — haven't yet optimised their AI visibility. The early positions in AI models are being claimed right now. Acting now means taking the lead.
3. Direct bookings via AI agents
ChatGPT (April 2025), Perplexity (March 2025) and soon Google Gemini allow hotel bookings directly from the AI interface. These features only work with hotels whose data is accessible to AI bots.
4. Amplification without SEO impact
AI search bots (OAI-SearchBot, Claude-SearchBot) don't affect your Google rankings. You create a new visibility channel with no risk to your existing performance.
Arguments for Restricting Training Bots
1. Content usage control
GPTBot, ClaudeBot and CCBot collect your content to integrate into model training data. Your text becomes permanent model weights, without direct attribution or compensation.
2. Commercial confidentiality
Your pricing strategies, differentiating descriptions, sales arguments — elements you may prefer not to feed into models used by indirect competitors.
3. Server load
The most aggressive bots can generate significant load. Recognised bots (GPTBot, ClaudeBot) generally respect Crawl-Delay and are responsible crawlers.
Recommended robots.txt Configuration for Hotels
Our recommendation: allow search bots, evaluate case by case for training bots.
# AI search bots — allow for AI visibility
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
# Training bots — your choice
# Option A: Allow (recommended for independent hotels)
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Google-Extended
Allow: /
# Option B: Block (if you prefer to protect your content)
# User-agent: GPTBot
# Disallow: /
# User-agent: ClaudeBot
# Disallow: /
# Other AI bots
User-agent: Applebot-Extended
Allow: /
User-agent: Amazonbot
Allow: /
# Default rule
User-agent: *
Allow: /
# Protect sensitive areas
User-agent: *
Disallow: /admin/
Disallow: /booking-widget/
Sitemap: https://www.your-hotel.com/sitemap.xmlImportant: robots.txt is a trust system, not a technical lock. Reputable bots (GPTBot, ClaudeBot, PerplexityBot) respect robots.txt. Malicious bots ignore it. For real technical protection, use server-side access restrictions.
Check Your Current Configuration
To find out whether your AI bots are currently blocked or allowed, type https://www.your-hotel.com/robots.txt in your browser and look for entries for GPTBot, ClaudeBot, PerplexityBot and Google-Extended.
Or use AIscore — the tool automatically checks your access status for all major AI bots and gives you a score with recommendations.
Are your AI bots configured correctly?
AIscore checks your robots.txt, schema, llms.txt and 88 other signals — free, in under 30 seconds.
Check my configuration →