ClaudeBot: what it is, what it reads, and whether to block it

The short answer

Anthropic documents all three bots on one help page, and says each honours robots.txt (Anthropic Help Centre, updated 7 April 2026). Most advice online treats "ClaudeBot" as one switch. It is three switches with very different costs. Across Cloudflare's network, user-initiated AI fetches rose from 2% to 3.2% of AI crawling between July 2024 and July 2025 (Cloudflare, August 2025). Small, and growing. That is the traffic a blanket block turns away.

What is ClaudeBot?

ClaudeBot is the user agent Anthropic uses to collect publicly available web content that may contribute to training its Claude models. It respects robots.txt and the non-standard Crawl-delay directive. Restricting it tells Anthropic to exclude your future content from training datasets. It does not answer questions or send visitors.

Anthropic puts it plainly: when a site restricts ClaudeBot, "it signals that the site's future materials should be excluded" from training (Anthropic). Note the word future. Blocking today does not remove anything already collected.

One naming trap. If you searched "claudebot" you may have meant Clawd Bot, a similarly named personal assistant that runs on your own machine and currently crowds the search results for this word. It is not one of Anthropic's documented crawlers. This article is about Anthropic's bots.

What do Claude-User and Claude-SearchBot do?

Claude-User visits a page when a person using Claude asks something that needs it, such as "what does this supplier charge". Claude-SearchBot crawls ahead of time to build the index behind Claude's search answers. One acts on a single person's request in the moment. The other decides whether you are in the pool Claude draws from.

Anthropic's current page names only these three. If your robots.txt carries rules for older names such as anthropic-ai or Claude-Web, they are not on that list, so do not rely on them to control the three bots above. Name each one.

Why we treat these bots as the early shape of agent traffic

Our view: Claude-User is not really a crawler. It is a customer's assistant reading your site on their behalf. As more people hand research and shortlisting to assistants, a growing share of your "visitors" will be software acting for a buyer, and the page it reads is the page that wins or loses the enquiry.

The facts behind that view are still early, and we will not dress them up.

Anthropic now sells an assistant that browses for people. Claude in Chrome became generally available on every paid Claude plan on 26 August 2026, and it can read pages, click links and type into forms (Claude blog). Cloudflare now has a category for exactly this: an agent is automated behaviour "acting, usually in real time, on a person's behalf" (Cloudflare, July 2026).

So here is where we think this goes. Today a buyer types a question, reads ten tabs and fills in your form. Increasingly, the assistant reads the tabs, compares suppliers and hands the person a shortlist. A site that blocks Claude-User is not protecting its content. It is turning away a future customer's assistant at the door. That is post-click optimisation with a new kind of visitor, and it starts with being let in.

Is ClaudeBot taking more than it gives back?

For training, yes, by Cloudflare's measure. Anthropic crawled about 38,000 pages for every visit it referred in July 2025, far more than OpenAI or Perplexity. Cloudflare says this may overstate the gap, because traffic from Claude's app carries no referrer. The imbalance is a fair reason to block training. It is not a reason to block search or user fetches.

The ratio fell 86.7% between January and July 2025 (Cloudflare), and Cloudflare notes that "traffic referred by Claude's native app does not include a Referer: header" (Cloudflare, July 2025). Treat the number as directional. Then decide training on its own merits: your content, your licensing, your appetite. It is a legitimate choice either way.

How to block ClaudeBot and keep Claude search

Give each bot its own group in robots.txt. Disallow ClaudeBot, and allow Claude-User and Claude-SearchBot explicitly, so a leftover wildcard rule cannot catch them. Repeat this on every subdomain. Do not block Anthropic's IP addresses instead: Anthropic warns that can stop its bots reading your opt-out at all.

If ClaudeBot's load is the problem rather than training, Anthropic supports Crawl-delay for ClaudeBot as a gentler option. Anthropic also publishes its crawler IP list at claude.com/crawling/bots.json, for verifying that a visit claiming to be Claude really is.

Is Cloudflare blocking Claude on your site?

Possibly, without anyone choosing it. Since 1 July 2025, every new domain on Cloudflare is asked during sign-up whether to allow AI crawlers. From 15 September 2026, new domains also block training and agent bots by default on pages that show ads, while search bots stay allowed. A firewall decision overrides robots.txt.

Cloudflare's press release on the first change says it would "block AI crawlers accessing content without permission or compensation, by default" (Cloudflare, 1 July 2025). The 2026 defaults are set out in Cloudflare's own post. Most B2B service sites show no ads, so the 2026 default may not touch them. The sign-up question and the one-click "Block AI bots" setting can. So can a hosting firewall that challenges anything automated.

How to check what your site allows

Read your robots.txt for named groups and for the wildcard, then confirm your firewall lets the bots through. The quickest route is our AI crawler access checker, which reads your robots.txt for 20 AI bots, including all three of Anthropic's, and flags when a firewall turns bots away first.

We hold ourselves to the same rule. Our own robots.txt names ClaudeBot, Claude-User and Claude-SearchBot and allows all three, on the reasoning above: we want assistants to be able to describe PostCliq accurately. Our own crawler follows the same courtesy, and how it behaves is documented on the PostCliqBot page.

What to do next

1. Open yoursite.com/robots.txt today. Look for groups naming ClaudeBot, Claude-User and Claude-SearchBot, and for a wildcard group that blocks everything.
2. Make the training decision deliberately. Block ClaudeBot if you do not want your content in training data. Leave the other two allowed.
3. Check Cloudflare or your host's firewall for AI bot blocking or challenges, and line it up with what robots.txt says.
4. Run the AI crawler access checker to confirm the verdict for each bot.
5. Make the page worth reading once the assistant is in. Our guide to getting cited by ChatGPT covers the content side, and generative engine optimisation covers the wider strategy.
6. Set a baseline with the monthly prompt test in how to check your AI visibility.

Frequently asked questions

Should I block ClaudeBot?

Block ClaudeBot if you do not want your future content used to train Anthropic's models. That is a fair business choice, and it does not stop Claude answering questions about you, because Claude-User and Claude-SearchBot are separate. If you sell services and want to be recommended, allow those two. Many businesses will choose to allow all three.

Does blocking ClaudeBot remove my site from Claude?

No. ClaudeBot only covers training. Claude-User fetches pages when a user asks, and Claude-SearchBot indexes pages for Claude's search. Each has its own robots.txt group. Blocking ClaudeBot alone leaves Claude able to read and cite your site, unless a wildcard rule or a firewall setting blocks the other two as well.

What is the ClaudeBot user agent?

The robots.txt token is ClaudeBot. Anthropic's other two tokens are Claude-User and Claude-SearchBot. Use those exact names in robots.txt groups. To confirm a visit really came from Anthropic rather than an impostor using the name, compare its IP address with the list Anthropic publishes at claude.com/crawling/bots.json.

Is ClaudeBot the same as Clawd Bot?

No. ClaudeBot is Anthropic's training crawler, which visits websites. Clawd Bot is a similarly named personal assistant that runs on your own computer, and it is not one of the crawlers Anthropic documents. The names are close enough that both appear in search results for "claudebot", but only Anthropic's bot reads your robots.txt.