Who is fetching, and for what
| Crawler | Operator | Feeds |
|---|---|---|
| GPTBot | OpenAI | Model training corpus (distinct from ChatGPT's browsing fetches) |
| OAI-SearchBot / ChatGPT-User | OpenAI | Search/browsing retrieval - the fetches behind cited answers |
| PerplexityBot | Perplexity | Retrieval index for answer generation |
| ClaudeBot | Anthropic | Crawling for Claude's systems |
| Google-Extended | Controls AI training use - separate from Googlebot, which still feeds Search and AI Overviews |
Names and behaviors evolve; verify against each operator's current documentation before finalizing a policy - and trust your logs over any list, including this one. Two more worth knowing by name: CCBot (Common Crawl's nonprofit crawler, reused as training data by multiple labs) and Bytespider (ByteDance's crawler, reported by third parties as inconsistent about robots.txt compliance). Deeper, dedicated reads on the two OpenAI bots and Anthropic's crawler: OAI-SearchBot and ClaudeBot.
What blocking actually does - and does not
Blocking retrieval bots removes you from the candidate pool for cited answers: for a business that wants customers to find it, that is self-inflicted invisibility. Blocking training-only bots is a narrower call - a legitimate licensing position for content businesses, with little effect on answer citations that run through retrieval fetches. The common mistake is one rule for both jobs: training use and retrieval use are separate decisions, and the bots are separable in robots.txt.
A sane default policy for a commercial site
- Allow retrieval/search bots - being citable is the point of publishing.
- Decide training bots on business grounds - for most service businesses, allowing costs nothing and widens reach; for paid-content businesses, blocking is a defensible stance.
- Never rely on the default - write the per-bot rules explicitly, so the policy survives the next robots.txt edit.
- Monitor the logs - policy is what you wrote; reality is what got fetched. Our crawler-log tooling watches the delta on our own estate daily.
Field note from our own logs: AI bot traffic patterns shift far faster than documentation updates. A policy set once and never monitored is a policy in name only.