# robots.txt: AI Crawler Access for Shopify > Your Shopify robots.txt may block ChatGPT and Perplexity from seeing your products. How to separate search, user, training, and ads-validation bots. - Canonical HTML: https://verityscore.io/en/kb/robots-crawlers/ - Markdown alternate: https://verityscore.io/en/kb/robots-crawlers.md - Language: en - Content type: kb - Published: 2026-01-28 - Updated: 2026-05-07 - Tags: robots-txt, crawlers, gptbot, perplexitybot - Audit zone: Robots.txt ## Why robots.txt is critical Your `robots.txt` file configuration directly impacts your store's accessibility to AI crawlers. A misconfigured robots.txt can make your product pages harder to retrieve, cite, validate, or recommend. ## Why it matters for AI The robots.txt file is the first thing a crawler visits before exploring your site. If it contains a blocking rule, a respectful crawler stops immediately. The important part is role separation: search bots, user-action fetchers, training bots, and ads-validation bots do not have the same business impact. ## AI Crawlers to Know | Crawler | Operator | Usage | |---------|----------|-------| | `GPTBot` | OpenAI | ChatGPT training and general AI knowledge | | `OAI-SearchBot` | OpenAI | ChatGPT search and citation (no training data collected) | | `OAI-AdsBot` | OpenAI | ChatGPT ads landing-page validation and relevance | | `ChatGPT-User` | OpenAI | ChatGPT with real-time browsing | | `PerplexityBot` | Perplexity | Perplexity Search & Shopping | | `ClaudeBot` | Anthropic | Claude with browsing (traffic doubled Q3 2025 - Q1 2026, [SE Ranking, 2026](https://seranking.com/blog/ai-statistics/)) | | `Google-Extended` | Google | AI usage/training control, not Google Search crawling | | `Amazonbot` | Amazon | Amazon search & Alexa | | `Bytespider` | ByteDance | TikTok AI features | **Why OAI-SearchBot matters**: Unlike GPTBot (used for training), OAI-SearchBot is OpenAI's retrieval crawler for search and shopping discovery. `OAI-AdsBot` is different again: it validates ad landing pages and should be treated as paid-media readiness, not organic GEO. ## Most Common Mistakes
robots.txt decision tree: AI crawler arrives, checks if a global Disallow blocks everything (site invisible), then if a specific block targets it (partially blocked), otherwise the site is accessible and indexed
Figure 1 - How an AI crawler decides to index your store based on robots.txt
1. **Global disallow** : `Disallow: /` blocks everything for all crawlers 2. **Specific AI bot blocking** : Some SEO guides recommend blocking AI crawlers. Counterproductive if you want recommendations. 3. **Confusion** : `Disallow: /` (blocks everything) vs `Disallow: /policies/` (blocks only policies) ## Recommended Configuration ``` User-agent: * Disallow: /admin Disallow: /cart Disallow: /orders Disallow: /checkouts/ Disallow: /account # OpenAI crawlers (training + search/citation) User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / # Other AI crawlers User-agent: PerplexityBot Allow: / User-agent: ClaudeBot Allow: / User-agent: Google-Extended Allow: / Sitemap: https://your-store.com/sitemap.xml ``` --- ## Related articles - [llms.txt for Shopify: Useful AI Discovery File, Not a Primary Ranking Signal](/en/kb/llms-txt) - [Agent Readiness Score: the 11 /.well-known/ files to publish in 2026](/en/blog/well-known-agent-ready) - [Understanding Your GEO Score: 9 Factors Explained](/en/kb/geo-readiness) - [GEO vs SEO: What's the Difference for E-commerce?](/en/kb/geo-vs-seo) - [Schema.org Product: Why and How on Shopify](/en/kb/schema-org) - [Sell on ChatGPT: The Complete Shopify Guide for 2026](/en/kb/sell-on-chatgpt-shopify) --- **Ready to check your store?** [Run a free GEO audit →](https://verityscore.io) ## FAQ ### Should I block or allow AI crawlers like GPTBot in robots.txt? Separate the roles. For organic discovery, avoid blocking search/user agents such as OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, and Claude-User. Training bots such as GPTBot are a separate consent decision, and OAI-AdsBot matters only for paid ChatGPT ads validation. ### How do I edit robots.txt on Shopify? Shopify generates robots.txt automatically, but you can customize it in your theme's robots.txt.liquid file (Online Store → Themes → Edit code → Templates → robots.txt.liquid). Add specific Allow rules for AI crawlers there. ### What is the difference between GPTBot and OAI-SearchBot? GPTBot is OpenAI's training crawler. OAI-SearchBot supports ChatGPT search and shopping discovery. ChatGPT-User fetches pages on behalf of a user, and OAI-AdsBot validates submitted ad landing pages. Do not treat them as one crawler. ### Does blocking AI bots improve my site security? No. Crawlers only view public pages, just like Googlebot. Blocking them does not secure private data. The business impact depends on the role: blocking search/user agents hurts organic discovery, while blocking training or ads-validation bots has a different meaning. ### What is the difference between robots.txt Disallow and blocking AI crawlers? A general 'Disallow: /' blocks all crawlers including Google. A specific bot block only blocks that one agent. The recommended approach is to allow public product and content pages for useful discovery agents while blocking sensitive paths like /admin and /cart. ## Sources - [70+ AI Search Stats for 2026 (Fully Verified & Up-to-Date)](https://seranking.com/blog/ai-statistics/) (industry) - [Q1 2026 AI Citation Trends Report for Shopify Brands](https://ecommercefastlane.com/2026-ai-citation-trends-report/) (industry)