# Blocked AI Crawlers: the Exact Audit Procedure
> Measured across 432 Shopify stores: 0.2% block an AI search crawler. The procedure to check you are not the exception, from robots.txt to your CDN.
- Canonical HTML: https://verityscore.io/en/blog/check-blocked-ai-crawlers-store/
- Markdown alternate: https://verityscore.io/en/blog/check-blocked-ai-crawlers-store.md
- Language: en
- Content type: blog
- Published: 2026-07-23
- Updated: 2026-07-24
- Tags: robots-txt, crawlers, cloudflare, gptbot, oai-searchbot, perplexitybot, claudebot, shopify, cdn
## The problem in 60 words
Blocked AI crawlers are far rarer than the internet suggests. We measured 432 French Shopify stores: **exactly one blocks an AI search crawler or a shopping agent**. Shopify ships a permissive `robots.txt` by default, and it takes a manual change to break it. So this procedure serves one purpose: confirming in 15 minutes that you are not the exception, and covering non-Shopify sites sitting behind a CDN.
### What we measured
On July 24, 2026, we read the `robots.txt` of 432 French Shopify stores (agency clients listed in the Shopify partner directory).
| Measure | Result |
|---|---|
| Name at least one AI crawler in their `robots.txt` | 9 / 432 (2.1%) |
| Block a training crawler (GPTBot, ClaudeBot) | 4 / 432 (0.9%) |
| Block an AI search crawler or a shopping agent | **1 / 432 (0.2%)** |
| Global `Disallow: /` block | 0 / 432 |
**Method**: read the `robots.txt` served with HTTP 200, parse `User-agent` groups, count a block when `Disallow: /` applies to the bot (its own group, or `*` as fallback). **Limits**: French sample, stores working with an agency (so probably better maintained than average), and the file layer only — a CDN or WAF block is invisible from the outside, which is exactly what step 3 covers.
Figure 1: the 4-step procedure. The CDN layer (steps 3-4) is invisible in robots.txt: that is where the blocks nobody decided are hiding.
## Step 0: understand the three crawler families
"AI bots" as a single category does not exist. There are three families, with three different business impacts. Everything else in this procedure follows from this. It is also the first building block of any GEO (Generative Engine Optimization) strategy: before optimizing your content for AI visibility, make sure the engines can simply read you.
| Family | User-agents | What they do | If you block them |
|---|---|---|---|
| Training crawlers | `GPTBot`, `ClaudeBot`, `Google-Extended` | Collect content to train models | Defensible choice: you give without getting |
| AI search crawlers | `OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot` | Index your store for ChatGPT, Claude, Perplexity answers | You disappear from those answers |
| Shopping agents | `ChatGPT-User`, `Claude-User`, `Perplexity-User` | Fetch a page when a real user asks a question | You shut the door on a customer mid-purchase |
Every operator documents this separation: [OpenAI](https://developers.openai.com/api/docs/bots), [Anthropic](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) and [Perplexity](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) all state that each crawler is controlled independently: blocking GPTBot does not affect OAI-SearchBot, and vice versa. Perplexity explicitly states that PerplexityBot is not used to train models.
Figure 2: the three AI crawler families and the cost of blocking each. Every family is controlled separately.
## Step 1: read your robots.txt (2 minutes)
**Exact path**: open `https://your-store.com/robots.txt` in a browser.
Look for these three patterns, from most severe to most sneaky:
**1. The global block**:
```
User-agent: *
Disallow: /
```
The whole site is blocked for every crawler. Rare but fatal.
**2. Targeted anti-AI blocks**:
```
User-agent: GPTBot
Disallow: /
User-agent: PerplexityBot
Disallow: /
```
Often added in 2023-2024 following "protect your content from AI" guides, then forgotten. Check every `User-agent` named: if it belongs to the search or shopping-agent family (table above), the block is costing you visibility and sales.
**3. Partial blocks**: `Disallow: /products/` or `Disallow: /collections/` under an AI user-agent. More discreet, same effect on product pages.
**Special case: a block wrapped in `# BEGIN Cloudflare Managed content` and `# END Cloudflare Managed Content`.** Those lines do not come from your site: this is Cloudflare's managed robots.txt, prepended to yours at the edge. Editing your origin file (Shopify or otherwise) will not remove them: the configuration lives in the Cloudflare dashboard (see step 3).
**Expected result**: on a Shopify store without customization, you should only see `Disallow` rules on `/admin`, `/cart`, `/orders`, `/checkouts/`, `/account` and similar. No `User-agent` block naming an AI crawler. If that is your case, skip straight to step 3: your robots.txt is healthy.
## Step 2: fix it on Shopify (5 minutes)
Shopify blocks no AI crawler by default. If your robots.txt contains AI blocks, they come from a custom `robots.txt.liquid` file in your theme.
**Exact path**: `Online Store` → `Themes` → `…` button on the live theme → `Edit code` → `Templates` folder → `robots.txt.liquid`.
- **If the file does not exist**: Shopify serves its default robots.txt, which is healthy. Nothing to do.
- **If the file exists**: find the `User-agent` blocks targeting AI crawlers and remove them. Keep the `robots.default_groups` Liquid loop that generates the default rules: Shopify recommends keeping these Liquid objects, because the default rules are updated automatically.
**Known limit**: Shopify classifies editing `robots.txt.liquid` as an unsupported customization. Shopify Support will not help edit this file, and one badly written rule can cut all your traffic. If you are not comfortable, delete the whole template (your theme falls back to Shopify's healthy default) rather than editing rule by rule.
#### Timeline after the fix
| Moment | What happens | What you see |
|---|---|---|
| T+0 | The new robots.txt is served immediately | `your-store.com/robots.txt` reflects the change on reload |
| T+0 to 24h | Each crawler re-reads the file on its next visit (most cache robots.txt up to 24h) | Nothing visible: normal propagation window |
| T+24h to a few days | AI search crawlers resume exploring the unblocked pages | Progressive reappearance in AI answers, citation never guaranteed |
**Check while waiting**: reload `your-store.com/robots.txt` in a private window to confirm the served version is the new one (not a cache).
## Step 3: check the CDN layer (5 minutes)
This is the step almost everyone skips, and it is where the blocks nobody decided are hiding. A CDN can refuse a crawler before it even reaches your robots.txt: that block is invisible in the file.
**Case 1: standard Shopify store.** Your CDN is managed by Shopify, which does not block AI crawlers by default. If steps 1 and 2 are clean, you are done. Watch out, however, for "content protection" or anti-bot apps installed on the store: check their configuration one by one.
**Case 2: site behind Cloudflare** (custom site, WooCommerce, headless, or a domain proxied through Cloudflare). Since July 1, 2025, Cloudflare blocks known AI crawlers by default on new domains. If your domain was added after that date, you are probably blocking without having chosen to.
**And from September 15, 2026, it expands.** Cloudflare announced new defaults on July 1, 2026: on new domains, the `Training` AND `Agent` categories will be blocked by default on pages that display ads (the `Agent` category explicitly includes ChatGPT-User and browser-use agents, i.e. shopping agents). `Search` stays allowed. Existing customers must set their preferences in Security settings before September 15.
**The trap you absolutely need to know**: the new defaults enforce the most restrictive applicable rule. Cloudflare states it in plain words: multi-purpose crawlers such as Googlebot, Applebot and BingBot will be blocked for customers who select to block `Training`. In other words, ticking "block AI training" at the Cloudflare level can remove you from classic Google Search. If you block training, do it in robots.txt (Google-Extended, GPTBot), not through the Cloudflare category, as long as this behaviour is in force.
**Exact path**: Cloudflare dashboard → your domain → `AI Crawl Control` → `Crawlers` tab.
- Every AI crawler appears with an `Action` column: select `Allow` or `Block` bot by bot.
- Available on the Free plan (user-agent detection; paid plans add Bot Management's advanced detection).
- Since July 1, 2026, Cloudflare also classifies verified bots by intent (`Search`, `Agent`, `Training`, `Transact`…), which maps exactly to the three families in step 0.
**Cloudflare's managed robots.txt.** If your robots.txt shows a `# BEGIN Cloudflare Managed content` block, the managed robots.txt feature is on. What it does, precisely: it prepends a `Content-signal: search=yes, ai-train=no` line and `Disallow: /` rules targeting eight training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Amazonbot, Applebot-Extended, meta-externalagent) to your file. It touches neither AI search crawlers nor shopping agents: that is consistent with the three-family logic, and a defensible setting if you refuse training.
**Exact path to enable or disable it**: `Security` → `Bots` → `Configure Bot Fight Mode` → `Instruct bot traffic with robots.txt` toggle (also reachable via `Security` → `Settings`, `Bot traffic` filter). Available on all plans. If the block persists after disabling, purge the Cloudflare cache for `/robots.txt`: the managed version can keep being served for a few minutes.
**Setting consistent with the three families**: allow AI search crawlers and shopping agents; decide freely on training crawlers.
#### Timeline after a Cloudflare change
| Moment | What happens | What you see |
|---|---|---|
| T+0 to 1 min | The rule propagates across the Cloudflare network | The `Crawlers` tab shows the new action |
| T+a few hours | Allowed crawlers resume visiting | The AI Crawl Control `Metrics` tab shows allowed requests per crawler |
**Check while waiting**: the AI Crawl Control `Metrics` tab lists recent requests per crawler, with their status. That is the direct proof: if `PerplexityBot` still shows `Blocked` after your change, the rule is wrong.
## Step 4: the final test
1. `your-store.com/robots.txt`: no remaining block targeting an AI search crawler or a shopping agent.
2. If Cloudflare: AI Crawl Control `Metrics` tab showing `Allowed` requests for family 2 and 3 crawlers.
3. Real agent-side test: ask ChatGPT (with search enabled) or Perplexity a question about one of your products, naming your brand. If your product page is accessible, the tool can fetch it live. One failed attempt proves nothing (citation depends on many other factors), but a systematic refusal to read a URL you paste is a signal of a remaining block.
## Troubleshooting
- **robots.txt does not change after editing the template** → you edited an unpublished theme. Check you were on the live theme, or republish.
- **robots.txt is clean but a crawler never shows up in logs/metrics** → block at another layer: anti-bot app, WAF, server firewall. On Cloudflare, also check custom WAF rules and Bots settings beyond AI Crawl Control.
- **A contractor tells you blocking all AI "protects your content"** → true for a publisher that lives off its content, false for a store that lives off being found and recommended. Blocking is decided family by family, not wholesale.
---
## Related articles
- [Shopify robots.txt: useful AI bot access](/en/kb/robots-crawlers/)
- [llms.txt for Shopify: a useful AI content index](/en/kb/llms-txt/)
- [Agent Readiness Score: the 11 /.well-known/ files to publish in 2026](/en/blog/well-known-agent-ready/)
- [The real AI traffic funnel: crawl vs click](/en/blog/ai-traffic-funnel-crawl-vs-click/)
---
**Want the full check in 60 seconds?** The free audit reads your robots.txt, tests your product pages' accessibility and verifies the signals each AI engine can actually extract. [Run a free GEO audit →](https://verityscore.io)
## FAQ
### How do I know if my store blocks AI crawlers?
Two checks: first read your-store.com/robots.txt and look for User-agent blocks targeting GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot or their equivalents. Then check the CDN layer: on Cloudflare, the AI Crawl Control dashboard lists every AI crawler with its allowed or blocked status. robots.txt alone is not enough, because a CDN-level block is invisible in that file.
### Does Shopify block AI crawlers by default?
No. Shopify's default robots.txt only blocks admin, cart and checkout pages. If your Shopify store blocks an AI crawler, a custom robots.txt.liquid file was added to the theme, often by an app or an SEO contractor. The fix is removing those rules from the template.
### Should I block GPTBot?
It is a defensible choice, not a mistake. GPTBot collects content to train OpenAI models: blocking it affects neither your presence in ChatGPT search (OAI-SearchBot) nor user-triggered visits (ChatGPT-User). Each crawler is controlled separately in robots.txt.
### Why is blocking ChatGPT-User or Claude-User serious for a store?
Because behind every one of those visits there is a real human asking a purchase question. Blocking those user-agents means refusing entry to a customer asking about your catalog. Unlike training crawlers, there is no content-protection benefit: the page is fetched on demand, not mass-collected.
### Does blocking AI crawlers hurt my classic Google SEO?
No, if you block the right bots. Googlebot, the crawler behind Google Search and AI Overviews, is distinct from Google-Extended, the token that controls how your content is used for Gemini. Blocking Google-Extended changes neither your Google Search rankings nor your presence in AI Overviews. Blocking Googlebot itself, however, removes you from everything: never do it. Also beware at the Cloudflare level: from September 15, 2026, the new defaults enforce the most restrictive rule, and blocking the Training category there also blocks multi-purpose crawlers like Googlebot. Blocking training is safer done in robots.txt.
### My robots.txt contains a # BEGIN Cloudflare Managed content block: where does it come from?
That is Cloudflare's managed robots.txt: when the feature is on, Cloudflare prepends your file with a Content-signal line (search=yes, ai-train=no) and Disallow rules targeting training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Amazonbot, Applebot-Extended, meta-externalagent). Editing your site's file does not remove it: turn off the Instruct bot traffic with robots.txt toggle under Security, Bots, Configure Bot Fight Mode. This block only targets the training family, not AI search crawlers or shopping agents.
### My site runs through Cloudflare: am I affected by default blocking?
Yes, if your domain was added to Cloudflare after July 1, 2025: known AI crawlers are blocked by default there unless you chose otherwise during onboarding. Check the Cloudflare dashboard, AI Crawl Control section, Crawlers tab: each bot has an Allow or Block action you can change individually, including on the Free plan.
## Sources
- [Cloudflare: Verified bots, classified by behaviour (updated July 1, 2026)](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/) (official)
- [Cloudflare: Your site, your rules, new AI traffic defaults on September 15, 2026 (July 1, 2026)](https://blog.cloudflare.com/content-independence-day-ai-options/) (official)
- [Cloudflare: managed robots.txt for AI crawlers (updated July 1, 2026)](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt) (official)
- [Cloudflare: AI Crawl Control, get started (updated April 23, 2026)](https://developers.cloudflare.com/ai-crawl-control/get-started/) (official)
- [Cloudflare blocks AI crawlers by default (press release, July 1, 2025)](https://www.cloudflare.com/press/press-releases/2025/cloudflare-just-changed-how-ai-crawlers-scrape-the-internet-at-large/) (official)
- [OpenAI: bots documentation (accessed July 23, 2026)](https://developers.openai.com/api/docs/bots) (official)
- [Anthropic: Claude crawlers and access control (accessed July 23, 2026)](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) (official)
- [Perplexity: crawlers documentation (accessed July 23, 2026)](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) (official)
- [Shopify: customize robots.txt with robots.txt.liquid (developer docs, 2026)](https://shopify.dev/docs/storefronts/themes/seo/robots-txt) (official)
- [Shopify Help Center: Editing robots.txt.liquid (2026)](https://help.shopify.com/en/manual/promoting-marketing/seo/editing-robots-txt) (official)
