Skip to content
Cloudflare Docs

Get started

This guide instructs you on how to:

  • View AI crawlers that are interacting with pages in your domain (a Cloudflare zone).
  • Create a rule to block AI crawlers on your pages.

Prerequisites

  1. Sign up for a Cloudflare account ↗.
  2. Connect your domain to Cloudflare.
  3. Make sure your domain is proxying traffic through Cloudflare.

1. Block all AI crawlers

To use AI Audit:

  1. Log in to the Cloudflare dashboard ↗, and select your account and domain.
  2. Go to AI Audit.
  3. From Most Popular Paths, select Block All.
  4. From the Bot traffic page, under Block AI Bots, select Enable.

This feature also enables Cloudflare's Bot Fight Mode: Block AI bots.

You can also create more complex rules when taking action on AI crawlers. For more information on creating more specific rules, refer to Create a custom rule in the dashboard .

Which bots will Cloudflare block?

When you enable this feature, Cloudflare will block bots that have been identified as AI Crawlers. These bots include, but are not limited to, the following bots:

  • Amazonbot (Amazon)
  • Applebot (Apple)
  • Bytespider (ByteDance)
  • ChatGPT-User (OpenAI)
  • ClaudeBot (Anthropic)
  • Claude-SearchBot (Anthropic)
  • Claude-User (Anthropic)
  • DuckAssistBot (DuckDuckGo)
  • Google-CloudVertexBot (Google)
  • GoogleOther (Google)
  • GPTBot (OpenAI)
  • Meta-ExternalAgent (Meta)
  • OAI-SearchBot (OpenAI)
  • PerplexityBot (Perplexity)
  • Perplexity-User (Perplexity)
  • PetalBot (Huawei)
  • TikTokSpider (ByteDance)

In addition to this list, all verified bots that are classified as AI Search, AI Assistant, AI Crawler, or an Archiver, as well as a number of unverified bots that fall into the verified bot categories are blocked. It does not block verified bots that fall into the Search Engine categories.

The bots classified in these categories may change in real time as new AI bots are detected.

If you are a bot operator and feel your bot may have been incorrectly categorized, add your bot to the list of verified bots ↗.

2. Block specific bot categories (Enterprise plan only)

Customers on the Enterprise plan -- and with a Bot Management subscription -- can choose to only block specific AI crawlers, while allowing others.

  1. Go to the AI Audit.
  2. From Most Popular Paths, select Block Some.
  3. From the Security rules page, select Create rule > Custom rules.
  4. Provide a name for the custom rule. For example, "Block unwanted AI crawlers".
  5. From the Field dropdown, select Verified Bot Category.
  6. From the Value dropdown, select the specific bot category you wish to block.
    • You can use And / Or buttons to add additional conditions. For example, you can use multiple Or options to include multiple bot categories in the same rule.
  7. From the Then take action... section:
    • For Choose action, select Block.
    • For With response type, select Default Cloudflare WAF block page.
  8. From the Place at section:
    • For Select order, select First.
  9. Select Save.

This custom rule will only block the AI bots which belong to the verified bot categories you have included in your rule (in step 6).

For more information on creating a custom WAF rule, refer to Create a custom rule in the dashboard.

3. Review detected AI crawlers

Review the AI crawlers detected on your site in the Metrics tab of the Cloudflare dashboard for key metrics.

Refer to Detect AI crawlers for more information.

Refer to the following related resources: