Blocking GPTBot stops OpenAI from using your content to train its models, but doesn't remove you from ChatGPT's search answers, which rely on a separate crawler, OAI-SearchBot. Most businesses that want AI visibility should allow search crawlers, then decide on training crawlers such as GPTBot and Google-Extended as a separate policy choice.
The main crawlers and tokens
AI companies use different crawlers for different jobs. Some fetch pages to show in search answers, some collect content to train models, and some only act when a user asks. Knowing which is which lets you make an informed choice.12
| Name | Operator | What it's for | If you block it |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Surfacing sites in ChatGPT's search features | Your site won't be shown in ChatGPT's search answers |
| GPTBot | OpenAI | Collecting content that may be used to train OpenAI's models | Your content isn't used for training; ChatGPT search is unaffected |
| ChatGPT-User | OpenAI | Fetching a page when a ChatGPT user's request needs it | Robots.txt rules may not apply, as the request comes from a user |
| Googlebot | Google Search, including AI Overviews and AI Mode | You disappear from Google Search | |
| Google-Extended | A control token for Gemini model training and grounding, not a separate crawler | No effect on Google Search inclusion or ranking | |
| PerplexityBot | Perplexity | Indexing sites for Perplexity's search results | Less visibility in Perplexity's answers |
A common set-up: visible in AI search, opted out of training
This robots.txt lets OpenAI's search crawler in while opting out of OpenAI and Google model training:
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /Check more than robots.txt
Some CDN and firewall services offer one-click AI bot blocking, which can block search crawlers as well as training crawlers, whatever your robots.txt says. Check those settings too, and look at your server logs to confirm which crawlers actually reach your site.
Our choice
We allow search and training crawlers on ufound.online, because being known to AI tools is central to our business. Publishers with valuable proprietary content may reasonably choose differently, and blocking training while allowing search is a sensible middle ground.
Questions, answered
Does blocking GPTBot affect our Google rankings?
No. GPTBot is OpenAI's crawler and has nothing to do with Google Search. Google uses its own crawlers.
Does blocking Google-Extended remove us from AI Overviews?
No. Google says Google-Extended doesn't affect a site's inclusion or ranking in Google Search, and AI Overviews are part of Google Search.23 It controls whether your content is used for Gemini training and grounding in some other Google products.
How quickly do robots.txt changes take effect?
It varies by crawler. OpenAI says it can take around 24 hours for its search systems to adjust after a robots.txt change.1 Other crawlers re-read robots.txt on their own schedules.
Sources 3
- 1Overview of OpenAI Crawlersdevelopers.openai.com
- 2Google's common crawlersdevelopers.google.com
- 3AI features and your websitedevelopers.google.com
All sources accessed September 2026.
