Should I let AI crawlers like GPTBot access my site?

Q: Should I let AI crawlers like GPTBot access my site?

In almost every case, yes. Blocking GPTBot, ClaudeBot, PerplexityBot, and Google-Extended keeps your pages out of what ChatGPT, Claude, Perplexity, and Gemini can retrieve and cite. If you want AI visibility, allow them. The exception is proprietary or paywalled content you never want reproduced.

Q: What is the difference between a training crawler and a retrieval crawler?

A training crawler like GPTBot or CCBot collects pages to build or update a model. A live-retrieval crawler like OAI-SearchBot or PerplexityBot fetches your page in the moment to answer a question, which is where citations come from. Blocking a retrieval crawler removes you from answers immediately; blocking a training crawler mainly affects future model knowledge.

Q: Does blocking Google-Extended remove me from Google Search?

No. Google-Extended is a Gemini and Vertex training opt-out token and is separate from Googlebot. Blocking it opts you out of Gemini training and grounding but keeps you in Google Search and AI Overviews, which are crawled by Googlebot. Blocking Googlebot removes you from all of it.

Q: Will allowing GPTBot make ChatGPT cite me?

Not on its own. Allowing GPTBot makes you eligible to be cited, not guaranteed. Whether you get named depends on whether your page is the clearest, most linkable answer to the query. Crawler access is one input, not a switch.

Q: Does robots.txt actually stop AI crawlers?

It stops compliant, named crawlers, which includes the major bots from OpenAI, Anthropic, Perplexity, and Google. Robots.txt is voluntary, so it will not stop scrapers that ignore it. For hard enforcement you need server-side blocking by user agent or IP.

Question

Should I let AI crawlers like GPTBot access my site?

Maciej Grabek · Accepted Answer

Yes, in almost every case. Blocking GPTBot, ClaudeBot, PerplexityBot, and Google-Extended keeps your pages out of what ChatGPT, Claude, Perplexity, and Gemini can retrieve and cite. If you want AI visibility, allow them. The exception is proprietary or paywalled content you never want reproduced. Blocking a training crawler and blocking a live-retrieval crawler have very different effects. The crawlers you are actually deciding about Robots.txt gives you per-agent control, so the real question is not "AI: yes or no" but which named bots to allow. The ones that matter for ChatGPT, Perplexity, Gemini, Claude, and DeepSeek visibility: GPTBot - OpenAI's crawler. Feeds training and grounding for ChatGPT. OAI-SearchBot and ChatGPT-User - OpenAI's retrieval and user-triggered browsing agents. These fetch pages live when ChatGPT search answers a question. ClaudeBot (and the older anthropic-ai token) - Anthropic's crawler for Claude. PerplexityBot and Perplexity-User - Perplexity's index crawler and its live fetch when it answers a query. Google-Extended - a Gemini and Vertex training opt-out token. It is not a real crawler and does not control Googlebot. CCBot - Common Crawl, the open dataset many models (including DeepSeek) train on. Training crawlers vs live-retrieval crawlers This is the distinction most robots.txt advice gets wrong. A training crawler collects pages to build or update a model. A live-retrieval crawler fetches your page in the moment, to answer a question the user just asked, and that is where citations come from. Blocking a training bot like GPTBot or CCBot mostly affects whether your content shapes future model knowledge. Blocking a retrieval bot like OAI-SearchBot or PerplexityBot has a sharper, immediate cost: the engine literally cannot open your page to cite it, even when it decides your page is the best answer. The Google case is the one people botch. Google-Extended and Googlebot are separate. Blocking Google-Extended opts you out of Gemini training and grounding but keeps you in Google Search and AI Overviews, because those are crawled by Googlebot. Blocking Googlebot removes you from all of it. If you want to appear anywhere Google surfaces answers, never block Googlebot. A robots.txt that allows AI visibility If visibility is the goal, the default is to allow the AI agents explicitly so there is no ambiguity: # Allow AI crawlers - training and retrieval User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Google-Extended Allow: / User-agent: CCBot Allow: / To do the opposite for one engine, swap that block to Disallow: / . Robots.txt is honored by the major named bots but it is voluntary, not a firewall - it stops compliant crawlers, not scrapers that ignore the file. For hard enforcement you need server-side blocking by user agent or IP. What blocking actually costs you Say an agency owner asks Perplexity: What is the best project management tool for marketing agencies? Perplexity answers by fetching a handful of live pages through PerplexityBot and citing them with numbered links. If your robots.txt disallows PerplexityBot, your page cannot be one of those citations, no matter how well it answers the question. You do not rank lower - you are simply not eligible. The same logic applies to ChatGPT search via OAI-SearchBot and to Google AI Overviews via Googlebot. When blocking still makes sense Allowing crawlers is the right call for most marketing, product, docs, and blog content. Block deliberately when: The content is paywalled or licensed and reproduction would cannibalize revenue. It is proprietary research or data you sell rather than publish. You have a legal or contractual reason to keep material out of training sets - in which case block the training bots (GPTBot, CCBot, Google-Extended) while leaving retrieval and search bots on, so you stay citable without feeding training. Be honest about the tradeoff: blocking training crawlers to protect content is legitimate, but it does not by itself win you visibility, and blocking retrieval crawlers actively removes you from the answers. Crawler access is one input, not a switch. Allowing GPTBot does not make ChatGPT cite you - it makes you eligible to be cited. Whether you actually get named still depends on whether your page is the clearest, most linkable answer to the query. The way to know is to run your key prompts through each engine and check whether you appear, then fix access first if you find you are blocked. avisibli tracks that across ChatGPT, Perplexity, Gemini, Claude, and DeepSeek so you can see whether allowing a crawler turned into real citations. avisibli is the GEO platform that publishes this answer library. Self-references are limited to topics where a tool-based answer is genuinely useful to readers.