Cloudflare's AI agent controls: which personal agents to let in

The short answer

What did Cloudflare announce on 4 November?

Cloudflare turned one blunt setting into a set of named choices. Until now, its Agents policy applied a single action to all agent traffic. The 4 November changelog renames the setting to AI Crawler and Agent access policies and lets you set an action for each of the top personal agents individually, under Security, then Settings.

The announcement on Cloudflare's blog describes three parts.

• Per-agent access policies. Set an overall Agents policy, then override it for individual personal agents. Each one can be Allow (do not block), Block on pages with ads, or Block. Search and Training policies stay separate and unchanged.
• Personal Assistants analytics. Bot analysis gains a Personal Assistants category beside Verified Bot, Likely Human and Likely Automated. A Top statistics widget lists which agents visit and how many requests each sends. Filter on one agent and the other widgets show its paths, status codes and networks, so you can see whether it is reading product pages, reaching checkout, logging in or hitting errors.
• Account Abuse Protection, in early access. Login and sign-up events show whether a personal agent made the request and which one. A fraud team can act on a single account through its hashed user ID without blocking that agent for every other customer.

Cloudflare's documentation says personal agents inherit the Agents policy by default, and that it recommends Allow (do not block) for them (Cloudflare docs). Neither the changelog nor the blog post names the agents in the list or says which plans include the analytics.

What is a personal agent, and how is it different from a crawler?

A personal agent acts in real time for one person, while a crawler collects pages for later. Cloudflare's documentation splits automated visitors into three groups, each with its own policy.

• Search crawlers index content to answer questions about it later.
• Agents act in real time for a person, including chat fetches and browser-use agents.
• Training crawlers take content to train or fine-tune a model, including mixed-purpose crawlers that also feed search.

The difference matters for your decision. A training crawler takes your content and may never send anyone back. A personal agent is usually there because a person asked it to find a supplier, compare prices or book something. Blocking it is closer to locking the door on a customer's assistant than to refusing a scraper. We set out how two large retailers chose opposite answers to the same question in our guide to whether to block AI agents.

Which setting should a business website choose?

For most business websites, Allow. If you sell services, take bookings or rely on enquiries, the agent is doing pre-sale work for a buyer. Blocking it removes you from the shortlist without anyone telling you. The other two settings exist for sites with different economics.

Two cautions before you change anything. First, check what your domain is actually set to. Cloudflare said in July that from 15 September 2026 new domains would block Training and Agent crawlers on pages with ads by default (Help Net Security). Its documentation today says new domains allow all three. The dashboard is the only reliable answer for your site. Second, an ad on a service page, such as a third-party review widget or a retargeting slot, may be enough for Block on pages with ads to apply where you did not expect it. Cloudflare does not explain how it detects ads.

What can the new analytics tell you?

It can show you, for the first time on Cloudflare, which assistants are working on your site and whether they succeed. That second part is the useful one. An agent that reaches your contact page and receives a 403 error is a lost enquiry that no conversion report will ever show.

The scale is already visible in other vendors' data, although none of it measures your site.

These are vendor figures reported through trade press, with samples drawn from each vendor's own customers. Read them as direction. The direction is that agents go deeper into a site than a casual human visitor, straight to the pages where money changes hands.

That makes the Personal Assistants view worth a weekly look. Filter on each agent, then read three things: the paths it requests, the status codes it gets back, and whether it ever reaches your enquiry or checkout page. A run of 403 or 429 responses on those pages points to a firewall or rate-limit rule that treats a customer's assistant as an attacker.

What is the Trusted Agent Delegation coalition?

It is Cloudflare's attempt to solve the problem the new switches cannot. In a 5 November post, Cloudflare argues that a website sees the agent, not the person behind it. It calls its answer the Intermediation Principle: the relationship between a person and a site, and each side's security responsibilities, should hold even when an agent sits between them.

The post lists 14 problems that follow when they do not, and says websites carry most of the cost. Among them:

• Traffic commingling. Block one abusive user's agent and you block every honest person using the same agent.
• Friendly fraud. A person approves a purchase through an agent, then disputes it, and the site has lost the evidence it would normally use.
• Identity spoofing. Agents copy a well-known user-agent string to borrow its allowances.
• User attribution. Agent traffic breaks personalisation, recommendations and loyalty tracking.

Cloudflare names Web Bot Auth, where an agent signs each request so a site can verify who sent it, as its preferred identity method. It sketches four levels of disclosure, from nothing about the person, through a session or an account, to a unique identifier across agents. The coalition's stated job is to bring operators, identity and payment companies, merchants and publishers together, review existing protocols and take defined problems to the IETF. It has not named its members. Our explainer on AI agent detection covers how Web Bot Auth works and which agents use it today.

Why does this matter now?

Because the assistants are arriving at the same moment the platforms pick sides. On 6 November, two days after Cloudflare's release, Amazon began letting invited US customers shop by text message with Alexa for Shopping, which can search, compare, build carts and complete purchases. Amazon says it will reach millions more customers in the coming weeks (PPC Land). In September, Amazon cut Meta's Muse off from buying on Amazon.com, citing among other things that the agent did not identify itself.

The pattern is clear. The largest platforms want their own agent to be the one customers use, and they set terms for everyone else's. A business website does not have that leverage. It has to be easy for whichever assistant the customer already trusts. Cloudflare's settings are the first time most site owners can see that traffic and choose deliberately instead of by default.

There is also a second door the agent has to pass. On WordPress, the same agent can clear Cloudflare and still be challenged by a security plugin's bot rule or a CAPTCHA on the form further down the stack, which is the layer a WordPress specialist such as WP Creative checks when enquiries stall without an obvious cause.

Where is this heading?

Cloudflare says agentic traffic may soon eclipse human browsing, without putting a date on it. That is a forecast, not a measurement, but the direction matches everything else in the data.

Our view: within a year, access policies for named agents will be as routine as robots.txt, and just as easy to get wrong. Most mistakes will not be deliberate blocks. They will be defaults nobody checked, an ad slot that changes a page's category, or a rate limit tuned for scrapers that also catches a customer's assistant. The sites that win will treat agent access as part of the conversion path. They will allow by default, read the analytics, and fix the specific pages where agents fail. We also expect identity to move from optional to expected. Agents that sign their requests and pass on what a site needs to know about the person will get trusted access. Agents that hide will be slowed down.

What does not change is what the agent needs once it is in: content in the HTML rather than built by JavaScript, clear facts about what you offer, and a contact path that works without a mouse. Chrome is already filling in forms with Gemini, so the form is now a machine interface whether you planned for it or not.

What to do next

1. Open Security, then Settings, then AI Crawler and Agent access policies in Cloudflare, and write down what your Search, Agents and Training policies are set to today.
2. Set personal agents to Allow unless you have a specific reason not to, such as ad revenue, scarce stock or account risk. Keep Training as a separate decision.
3. Check the Personal Assistants analytics weekly for each agent's paths and status codes, and look for 403 and 429 errors on service, pricing, contact and checkout pages.
4. Run the AI crawler access checker to see whether your robots.txt and firewall let AI crawlers and agents reach your pages.
5. Run the agent readiness check to confirm an agent can read your content once it is let in.
6. Test your contact path with can visitors contact you, then submit your own form from a cloud browser to see whether a CAPTCHA stops it.

Frequently asked questions

What are Cloudflare's AI Crawler and Agent access policies?

They are Cloudflare's settings for automated visitors, split into Search, Agents and Training, each with its own action. Since 4 November 2026 you can also set Allow, Block on pages with ads, or Block for each of the top personal agents individually. They sit under Security, then Settings, in the Cloudflare dashboard.

Should I block personal AI agents on Cloudflare?

For most business websites, no. A personal agent usually acts for someone comparing suppliers or trying to buy, so blocking it can cost you the enquiry. Cloudflare itself recommends Allow. Block selectively where an agent harms you, such as pages that depend on ad impressions or stock that sells out.

How do I see AI agent traffic in Cloudflare?

Open Security Analytics and select the Personal Assistants category in Bot analysis. The Top statistics widget lists each agent and its request volume. Filter on one agent to see the paths, status codes and networks it used, which shows whether it reached your contact or checkout pages.

What is Cloudflare's Trusted Agent Delegation?

It is a coalition Cloudflare announced on 5 November 2026 to work out how websites can know and trust the person an AI agent acts for. It favours Web Bot Auth for agent identity and plans to take defined problems to the IETF. It is not yet a technical standard, and no members have been named.