Should you block GPTBot and other AI crawlers?

Which AI crawlers to allow or block, and how to stay visible while opting out of training.

Written by Mohd Shahrukh, FounderLast reviewed 2 min read
Illustration for Should you block GPTBot and other AI crawlers?SEO
The short answer

Blocking GPTBot stops OpenAI from using your content to train its models, but doesn't remove you from ChatGPT's search answers, which rely on a separate crawler, OAI-SearchBot. Most businesses that want AI visibility should allow search crawlers, then decide on training crawlers such as GPTBot and Google-Extended as a separate policy choice.

Part of our guide toAI search optimisation

The main crawlers and tokens

AI companies use different crawlers for different jobs. Some fetch pages to show in search answers, some collect content to train models, and some only act when a user asks. Knowing which is which lets you make an informed choice.12

The main crawlers and tokens
NameOperatorWhat it's forIf you block it
OAI-SearchBotOpenAISurfacing sites in ChatGPT's search featuresYour site won't be shown in ChatGPT's search answers
GPTBotOpenAICollecting content that may be used to train OpenAI's modelsYour content isn't used for training; ChatGPT search is unaffected
ChatGPT-UserOpenAIFetching a page when a ChatGPT user's request needs itRobots.txt rules may not apply, as the request comes from a user
GooglebotGoogleGoogle Search, including AI Overviews and AI ModeYou disappear from Google Search
Google-ExtendedGoogleA control token for Gemini model training and grounding, not a separate crawlerNo effect on Google Search inclusion or ranking
PerplexityBotPerplexityIndexing sites for Perplexity's search resultsLess visibility in Perplexity's answers

Based on OpenAI's and Google's crawler documentation.12

A common set-up: visible in AI search, opted out of training

This robots.txt lets OpenAI's search crawler in while opting out of OpenAI and Google model training:

robots.txt example
User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Check more than robots.txt

Some CDN and firewall services offer one-click AI bot blocking, which can block search crawlers as well as training crawlers, whatever your robots.txt says. Check those settings too, and look at your server logs to confirm which crawlers actually reach your site.

Our choice

We allow search and training crawlers on ufound.online, because being known to AI tools is central to our business. Publishers with valuable proprietary content may reasonably choose differently, and blocking training while allowing search is a sensible middle ground.

Part of our guide toAI search optimisation

Questions, answered

Does blocking GPTBot affect our Google rankings?

No. GPTBot is OpenAI's crawler and has nothing to do with Google Search. Google uses its own crawlers.

Does blocking Google-Extended remove us from AI Overviews?

No. Google says Google-Extended doesn't affect a site's inclusion or ranking in Google Search, and AI Overviews are part of Google Search.23 It controls whether your content is used for Gemini training and grounding in some other Google products.

How quickly do robots.txt changes take effect?

It varies by crawler. OpenAI says it can take around 24 hours for its search systems to adjust after a robots.txt change.1 Other crawlers re-read robots.txt on their own schedules.

Was this helpful?

Sources 3

  1. 1Overview of OpenAI Crawlersdevelopers.openai.com
  2. 2Google's common crawlersdevelopers.google.com
  3. 3AI features and your websitedevelopers.google.com

All sources accessed September 2026.

Free checklist

Audit your website in an hour.

Related serviceTechnical SEO & SEO audit

Find and fix what stops Google ranking your site.

Explore →

Keep reading

Article · 2 min readHow ChatGPT decides which businesses to recommend
Article · 2 min readAEO vs GEO vs SEO: what's the difference?
Article · 1 min readDoes ChatGPT recommend my business? How to check

Check who can read your website.

Send us your website. We'll check your robots.txt, firewall settings and which AI crawlers can reach you.

You'll hear back from Mohd ShahrukhFounder, ufound · 9+ years in digital marketing