ClaudeBot: what it crawls, and how to control it
Anthropic's crawler shows up in more logs every month. Here is what it actually does, and the decision it forces on your robots.txt.
In short
ClaudeBot is Anthropic's web crawler, used to gather content for Claude's models. Its user-agent token is ClaudeBot, and the operator states it respects robots.txt. Whether to allow it is a training-data decision distinct from whether Claude, the assistant, can cite you when it browses live - those can run through different pathways.
What it is and who runs it
ClaudeBot is Anthropic's crawler, used to gather web content that feeds Claude's models - the same job category as GPTBot for OpenAI, run by a different operator.
ClaudeBot vs Claude's live browsing
The bulk training crawler and Claude the assistant's live browsing/citation behavior are not necessarily the same pathway, the same distinction that separates GPTBot from OAI-SearchBot on the OpenAI side. Don't assume one robots.txt rule governs both; verify current behavior in your own logs rather than assuming.
Should you allow it
The same training-vs-retrieval logic applies here as with any training-oriented crawler: most service businesses have little to lose and reach to gain by allowing it; content businesses with a real licensing stance may reasonably restrict it. Make it a deliberate, named decision rather than an inherited default.
Verifying it in your logs
Match the user-agent string, then cross-check the requesting IP against Anthropic's currently published ranges before trusting the label alone. Server or CDN logs, filtered by user-agent and verified by IP where the operator publishes ranges, is the same method that applies to every bot in this list - trust your logs over any fixed table, including this one.
Common questions
What does ClaudeBot use my content for?
Training data collection for Anthropic's Claude models, by the operator's own account - not necessarily the live retrieval that powers a specific chat answer.
Does blocking ClaudeBot stop Claude from citing my site?
Not necessarily - live citation in Claude's assistant product can run through different retrieval mechanisms than the bulk training crawler. Verify current behavior in logs rather than assuming one rule covers both.
How is ClaudeBot different from GPTBot?
Same job category - model-training crawl - different operator. GPTBot serves OpenAI's training corpus; ClaudeBot serves Anthropic's. Both are governed the same way: a training-use decision, separate from retrieval bots.
How do I confirm a hit is really ClaudeBot?
Match the user-agent string and cross-check the requesting IP against Anthropic's currently published ranges before trusting the label.