Skip to main content
Critical Robots.txt

robots.txt and AI Crawlers: Don't Block Your Sales

Updated Recently updated
#robots-txt#crawlers#gptbot#perplexitybot
Share
This article includes a quiz, test your knowledge at the end! Jump to quiz

Why robots.txt is critical

Your robots.txt file configuration directly impacts your store’s accessibility to AI crawlers. A misconfigured robots.txt can make your store completely invisible to AI agents.

Why it matters for AI

The robots.txt file is the first thing a crawler visits before exploring your site. If it contains a blocking rule, a respectful crawler (like GPTBot or ClaudeBot) stops immediately. Your store is never read, never indexed, never recommended.

AI Crawlers to Know

CrawlerOperatorUsage
GPTBotOpenAIChatGPT training and general AI knowledge
OAI-SearchBotOpenAIChatGPT search and citation (no training data collected)
ChatGPT-UserOpenAIChatGPT with real-time browsing
PerplexityBotPerplexityPerplexity Search & Shopping
ClaudeBotAnthropicClaude with browsing (traffic doubled Q3 2025–Q1 2026)
Google-ExtendedGoogleGemini, AI Overview
AmazonbotAmazonAmazon search & Alexa
BytespiderByteDanceTikTok AI features

Why OAI-SearchBot matters: Unlike GPTBot (used for training), OAI-SearchBot is OpenAI’s real-time retrieval crawler that powers ChatGPT’s search and shopping citation features. Allowing it ensures your product pages can appear directly cited in ChatGPT Shopping results. OpenAI updated its crawler documentation in December 2025 to clarify the distinct roles of each bot (source: OpenAI Developer Docs, December 2025).

Most Common Mistakes

robots.txt decision tree: AI crawler arrives, checks if a global Disallow blocks everything (site invisible), then if a specific block targets it (partially blocked), otherwise the site is accessible and indexed
Figure 1 - How an AI crawler decides to index your store based on robots.txt
  1. Global disallow : Disallow: / blocks everything for all crawlers
  2. Specific AI bot blocking : Some SEO guides recommend blocking AI crawlers. Counterproductive if you want recommendations.
  3. Confusion : Disallow: / (blocks everything) vs Disallow: /policies/ (blocks only policies)
User-agent: *
Disallow: /admin
Disallow: /cart
Disallow: /orders
Disallow: /checkouts/
Disallow: /account

# OpenAI crawlers (training + search/citation)
User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

# Other AI crawlers
User-agent: PerplexityBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Google-Extended
Allow: /

Sitemap: https://your-store.com/sitemap.xml

Ready to check your store? Run a free GEO audit →

Frequently Asked Questions

Should I block or allow AI crawlers like GPTBot in robots.txt?
If you want your products to be recommended by AI shopping agents (ChatGPT, Perplexity, Claude), you should allow them. Blocking AI crawlers means your store is invisible to those platforms entirely.
How do I edit robots.txt on Shopify?
Shopify generates robots.txt automatically, but you can customize it in your theme's robots.txt.liquid file (Online Store → Themes → Edit code → Templates → robots.txt.liquid). Add specific Allow rules for AI crawlers there.
What is the difference between GPTBot and OAI-SearchBot?
GPTBot is OpenAI's general-purpose crawler used for training and general AI knowledge. OAI-SearchBot is OpenAI's real-time retrieval crawler that powers ChatGPT's search and citation features - it does not collect training data. Both should be allowed for full ChatGPT visibility.
Does blocking AI bots improve my site security?
No. AI crawlers like GPTBot and ClaudeBot are read-only. They only view public pages, just like Googlebot. Blocking them doesn't improve security, it only reduces your store's visibility in AI-powered search and shopping.
What is the difference between robots.txt Disallow and blocking AI crawlers?
A general 'Disallow: /' blocks all crawlers including Google. Specific bot blocks (User-agent: GPTBot / Disallow: /) only block that one AI. The recommended approach is to allow AI crawlers while blocking sensitive paths like /admin and /cart.
🧠 Test your knowledge
1 / 5

Related guides

Optimization

llms.txt: The New robots.txt for LLMs

What is llms.txt, why add it to your Shopify store, and how to write it to guide AI agents toward your key content.

Critical

ChatGPT App for Shopify: Build Your Own Branded AI

How Shopify merchants can build a dedicated ChatGPT App with loyalty, guided selling, and brand voice. Step-by-step guide, costs, and real examples.

Critical

What is Agentic Commerce? Complete Guide 2026

Autonomous AI agents search, compare, and buy for consumers. Understand ACP, UCP, and how to prepare your Shopify store.