AI Audit enables you to take specific action for each AI crawler.
To manage AI crawlers:
Log in to the Cloudflare dashboard ↗ , and select your account and domain.
Go to AI Audit .
Go to the AI Crawlers tab.
Review AI crawler activity The AI Crawlers tab displays a table of AI crawlers that are requesting access to your content, and how they interact with your pages. The table provides the following information.
Column Details AI Crawlers The name of the AI crawler. Operator The name of the entity who owns the AI crawler. Note that an operator may have multiple AI crawlers. Category The category of the AI crawler. Refer to Verified bot categories . Block Toggle for blocking specific AI crawlers.
Column Details AI Crawlers The name of the AI crawler. Operator The name of the entity who owns the AI crawler. Note that an operator may have multiple AI crawlers. Category The category of the AI crawler. Refer to Verified bot categories . Requests The number of times the AI crawler has requested to crawl your content. Robots.txt violations The number of times the AI crawler has violated your robots.txt Action The action you wish to take for the AI crawler. Refer to Take action for each AI crawler .
On the free plan, AI Audit identifies AI crawlers based on their user agent strings ↗ . This enables AI Audit to detect easy-to-detect (well-known) AI crawlers.
Upgrade your plan to enable a more thorough detection using Cloudflare's Bot Management detection ID field.
You can use filters to narrow the scope of your result:
Name: Search the name of the AI crawler.Operator: Filter by the AI crawler operator.Category: Filter by the category of the AI crawler (for example, AI crawler, AI assistant, or archiver).
The values of the table will update according to your filter.
Take action for each AI crawler For each AI crawler, you can take one of three actions: allow, charge, or block.
Allow access
Summary: You can allow an AI crawler to scrape your content.When to use: Allow AI crawlers that offer services which provide value through citations, referrals, or existing agreements.Implementation: From the Actions column, select Allow .
Note that you can still choose to Enforce robots.txt . For more details on how this rule interacts with other Cloudflare settings, refer to How it works .
Block access
Summary: You can block an AI crawler to completely stop the AI crawler from scraping your webpage.When to use: Block AI crawlers when their behavior do not align with your content strategy, or violate your policies.Implementation: From the Actions column, select Block . Charge for crawl (private beta)
Summary: You can charge the owner of the AI crawler for each crawl request.When to use: Charge AI crawlers when your content has training value, and you want to explore monetization optionsImplementation: From the Actions column, select Charge . For more information, refer to What is Pay Per Crawl? .
For each AI crawler, you can choose to allow or block access.
Allow access
Summary: You can allow an AI crawler to scrape your content.When to use: Allow AI crawlers that offer services which provide value through citations, referrals, or existing agreements.Implementation: From the Actions column, select Allow .
Note that you can still choose to Enforce robots.txt . Block access
Summary: You can block an AI crawler to completely stop the AI crawler from scraping your webpage.When to use: Block AI crawlers when their behavior do not align with your content strategy, or violate your policies.Implementation: From the Actions column, select Block .
Use pay per crawl to charge AI crawlers every time they access your content.
