Skip to main content
Available on Shopify The Shopify app that audits and fixes what AI reads about your products. Verity Score is on Shopify Install app
Critical Robots.txt

robots.txt: AI Crawler Access for Shopify

Updated Recently updated
#robots-txt#crawlers#gptbot#perplexitybot
Share
This article includes a quiz, test your knowledge at the end! Jump to quiz

Why robots.txt is critical

Your robots.txt file configuration directly impacts your store’s accessibility to AI crawlers. A misconfigured robots.txt can make your product pages harder to retrieve, cite, validate, or recommend.

Why it matters for AI

The robots.txt file is the first thing a crawler visits before exploring your site. If it contains a blocking rule, a respectful crawler stops immediately. The important part is role separation: search bots, user-action fetchers, training bots, and ads-validation bots do not have the same business impact.

AI Crawlers to Know

CrawlerOperatorUsage
GPTBotOpenAIChatGPT training and general AI knowledge
OAI-SearchBotOpenAIChatGPT search and citation (no training data collected)
OAI-AdsBotOpenAIChatGPT ads landing-page validation and relevance
ChatGPT-UserOpenAIChatGPT with real-time browsing
PerplexityBotPerplexityPerplexity Search & Shopping
Perplexity-UserPerplexityFetch triggered by a direct user action
ClaudeBotAnthropicClaude with browsing (traffic doubled Q3 2025 - Q1 2026, SE Ranking, 2026)
Google-ExtendedGoogleAI usage/training control, not Google Search crawling
AmazonbotAmazonAmazon search & Alexa
BytespiderByteDanceTikTok AI features

Why OAI-SearchBot matters: Unlike GPTBot (used for training), OAI-SearchBot is OpenAI’s retrieval crawler for search and shopping discovery. OAI-AdsBot is different again: it validates ad landing pages and should be treated as paid-media readiness, not organic GEO.

PerplexityBot is not Perplexity-User: PerplexityBot serves Perplexity’s search results and is not used to train foundation models. Perplexity-User is triggered when someone asks Perplexity to visit a specific page, and it generally ignores robots.txt. Allowing one does not allow the other, and blocking PerplexityBot does not stop a user-initiated fetch. Separate the two questions: is the crawler allowed, and is the page reachable when a user asks for it. (Perplexity crawler docs)

What to check

robots.txt decision tree: AI crawler arrives, checks if a global Disallow blocks everything (site invisible), then if a specific block targets it (partially blocked), otherwise the site is accessible and indexed
Figure 1 - How an AI crawler decides to index your store based on robots.txt

Your robots.txt can contain several types of blocking rules:

  • Global block: Disallow: / under User-agent: * means everything is blocked for every crawler
  • Specific blocks: User-agent: GPTBot plus Disallow: / means this crawler is explicitly targeted and blocked
  • Partial blocks: Disallow: /policies/ means only your policy pages are blocked (less severe, still damaging)

If a crawler is not named explicitly, the rules under User-agent: * apply to it.

Verity Score analyzes your robots.txt automatically and flags any AI crawler you are blocking.

The Shopify default case

Shopify’s default robots.txt contains:

User-agent: *
Disallow: /admin
Disallow: /cart
Disallow: /orders
Disallow: /checkouts/
Disallow: /*/checkouts
Disallow: /carts
Disallow: /account

These rules are correct: they only block admin and checkout pages. Your product pages, collections and content pages stay accessible.

Watch out: some Shopify apps or custom setups add extra rules that can block AI crawlers. Check regularly.

Most Common Mistakes

Mistake 1: global disallow

User-agent: *
Disallow: /

This blocks everything on your site for every crawler. It is the worst possible configuration.

Mistake 2: blocking AI bots specifically

User-agent: GPTBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

Some guides recommend blocking AI crawlers to “protect your content”. It is counterproductive if you want these systems to recommend you.

Mistake 3: confusing Disallow: / with Disallow: /policies/

Disallow: / blocks the ENTIRE site. Disallow: /policies/ blocks only the /policies/ directory. The difference is a single character, the impact is total.

How to fix it on Shopify

Check your current robots.txt

Go to https://your-store.com/robots.txt and read the contents.

Edit it on Shopify

On Shopify, robots.txt is controlled through the robots.txt.liquid file in your theme:

  1. Shopify Admin, then Themes, then Actions, then Edit code
  2. Look for robots.txt.liquid in the Templates folder
  3. Confirm that no AI crawler is blocked
  4. If the file does not exist, Shopify serves the default (which is correct)
User-agent: *
Disallow: /admin
Disallow: /cart
Disallow: /orders
Disallow: /checkouts/
Disallow: /account

# OpenAI crawlers (training + search/citation)
User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

# Other AI crawlers
User-agent: PerplexityBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Google-Extended
Allow: /

Sitemap: https://your-store.com/sitemap.xml

Resources

Frequently Asked Questions

Should I block or allow AI crawlers like GPTBot in robots.txt?
Separate the roles. For organic discovery, avoid blocking search/user agents such as OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, and Claude-User. Training bots such as GPTBot are a separate consent decision, and OAI-AdsBot matters only for paid ChatGPT ads validation.
How do I edit robots.txt on Shopify?
Shopify generates robots.txt automatically, but you can customize it in your theme's robots.txt.liquid file (Online Store → Themes → Edit code → Templates → robots.txt.liquid). Add specific Allow rules for AI crawlers there.
What is the difference between GPTBot and OAI-SearchBot?
GPTBot is OpenAI's training crawler. OAI-SearchBot supports ChatGPT search and shopping discovery. ChatGPT-User fetches pages on behalf of a user, and OAI-AdsBot validates submitted ad landing pages. Do not treat them as one crawler.
Does blocking AI bots improve my site security?
No. Crawlers only view public pages, just like Googlebot. Blocking them does not secure private data. The business impact depends on the role: blocking search/user agents hurts organic discovery, while blocking training or ads-validation bots has a different meaning.
What is the difference between robots.txt Disallow and blocking AI crawlers?
A general 'Disallow: /' blocks all crawlers including Google. A specific bot block only blocks that one agent. The recommended approach is to allow public product and content pages for useful discovery agents while blocking sensitive paths like /admin and /cart.
🧠 Test your knowledge
1 / 5

Related guides

Critical

LLM Landscape: 15 AI Systems in 2026

Beyond ChatGPT and Perplexity: Mistral, Claude, Gemini, Meta AI, Grok, DeepSeek, and MCP agents. Who reads what, how to appear everywhere.

Critical

AggregateRating: Make Reviews AI-Readable

Are your Loox, Judge.me, Yotpo reviews readable by ChatGPT, Perplexity, Mistral? Analysis per app, per AI crawler, and server-side fix paths.

Optimization

llms.txt for Shopify: Worth Adding?

llms.txt maps your Shopify content for AI systems, but schema.org, sitemaps and HTML still drive more visibility. Learn what to prioritize.