Category: Uncategorized

AI Crawl Control now includes new tools to help you prepare your site for the agentic Internet—a web where AI agents are first-class citizens that discover and interact with content differently than human visitors.

Content Format insights

The Metrics tab now includes a Content Format chart showing what content types AI systems request versus what your origin serves. Understanding these patterns helps you optimize content delivery for both human and agent consumption.

Directives tab (formerly Robots.txt)

The Robots.txt tab has been renamed to Directives and now includes a link to check your site’s Agent Readiness score.

Refer to our blog post on preparing for the agentic Internet for more on why these capabilities matter.

Radar adds three new features to the AI Insights page, expanding visibility into how AI bots, crawlers, and agents interact with the web.

Adoption of AI agent standards

The AI Insights page now includes an adoption of AI agent standards widget that tracks how websites adopt agent-facing standards. The data is filterable by domain category and updated weekly on Mondays.
This data is also available through the Agent Readiness API reference.

URL Scanner reports now include an Agent readiness tab that evaluates a scanned URL against the criteria used by the Agent Readiness score tool.

For more details, refer to the Agent Readiness blog post.

Markdown for Agents savings

A new savings gauge shows the median response-size reduction when serving Markdown instead of HTML to AI bots and crawlers. This highlights the bandwidth and token savings that Markdown for Agents provides.

For more details, refer to the Markdown for Agents API reference.

Response status

The new response status widget displays the distribution of HTTP response status codes returned to AI bots and crawlers. Results are groupable by individual status code (200, 403, 404) or by category (2xx, 3xx, 4xx, 5xx).

The same widget is available on each verified bot’s detail page (only available for AI bots), for example Google.

Explore all three features on the Cloudflare Radar AI Insights page.

Cloudflare’s network now supports redirecting verified AI training crawlers to canonical URLs when they request deprecated or duplicate pages. When enabled via AI Crawl Control > Quick Actions, AI training crawlers that request a page with a canonical tag pointing elsewhere receive a 301 redirect to the canonical version. Humans, search engine crawlers, and AI Search agents continue to see the original page normally.

This feature leverages your existing <link rel="canonical"> tags. No additional configuration required beyond enabling the toggle. Available on Pro, Business, and Enterprise plans at no additional cost.

Refer to the Redirects for AI Training documentation for details.

AI Crawl Control now includes new tools to help you prepare your site for the agentic Internet—a web where AI agents are first-class citizens that discover and interact with content differently than human visitors.

Content Format insights

The Metrics tab now includes a Content Format chart showing what content types AI systems request versus what your origin serves. Understanding these patterns helps you optimize content delivery for both human and agent consumption.

Directives tab (formerly Robots.txt)

The Robots.txt tab has been renamed to Directives and now includes a link to check your site’s Agent Readiness score.

Refer to our blog post on preparing for the agentic Internet for more on why these capabilities matter.

Radar adds three new features to the AI Insights page, expanding visibility into how AI bots, crawlers, and agents interact with the web.

Adoption of AI agent standards

The AI Insights page now includes an adoption of AI agent standards widget that tracks how websites adopt agent-facing standards. The data is filterable by domain category and updated weekly on Mondays.
This data is also available through the Agent Readiness API reference.

URL Scanner reports now include an Agent readiness tab that evaluates a scanned URL against the criteria used by the Agent Readiness score tool.

For more details, refer to the Agent Readiness blog post.

Markdown for Agents savings

A new savings gauge shows the median response-size reduction when serving Markdown instead of HTML to AI bots and crawlers. This highlights the bandwidth and token savings that Markdown for Agents provides.

For more details, refer to the Markdown for Agents API reference.

Response status

The new response status widget displays the distribution of HTTP response status codes returned to AI bots and crawlers. Results are groupable by individual status code (200, 403, 404) or by category (2xx, 3xx, 4xx, 5xx).

The same widget is available on each verified bot’s detail page (only available for AI bots), for example Google.

Explore all three features on the Cloudflare Radar AI Insights page.

Cloudflare’s network now supports redirecting verified AI training crawlers to canonical URLs when they request deprecated or duplicate pages. When enabled via AI Crawl Control > Quick Actions, AI training crawlers that request a page with a canonical tag pointing elsewhere receive a 301 redirect to the canonical version. Humans, search engine crawlers, and AI Search agents continue to see the original page normally.

This feature leverages your existing <link rel="canonical"> tags. No additional configuration required beyond enabling the toggle. Available on Pro, Business, and Enterprise plans at no additional cost.

Refer to the Redirects for AI Training documentation for details.

AI Crawl Control now includes new tools to help you prepare your site for the agentic Internet—a web where AI agents are first-class citizens that discover and interact with content differently than human visitors.

Content Format insights

The Metrics tab now includes a Content Format chart showing what content types AI systems request versus what your origin serves. Understanding these patterns helps you optimize content delivery for both human and agent consumption.

Directives tab (formerly Robots.txt)

The Robots.txt tab has been renamed to Directives and now includes a link to check your site’s Agent Readiness score.

Refer to our blog post on preparing for the agentic Internet for more on why these capabilities matter.

Radar adds three new features to the AI Insights page, expanding visibility into how AI bots, crawlers, and agents interact with the web.

Adoption of AI agent standards

The AI Insights page now includes an adoption of AI agent standards widget that tracks how websites adopt agent-facing standards. The data is filterable by domain category and updated weekly on Mondays.
This data is also available through the Agent Readiness API reference.

URL Scanner reports now include an Agent readiness tab that evaluates a scanned URL against the criteria used by the Agent Readiness score tool.

For more details, refer to the Agent Readiness blog post.

Markdown for Agents savings

A new savings gauge shows the median response-size reduction when serving Markdown instead of HTML to AI bots and crawlers. This highlights the bandwidth and token savings that Markdown for Agents provides.

For more details, refer to the Markdown for Agents API reference.

Response status

The new response status widget displays the distribution of HTTP response status codes returned to AI bots and crawlers. Results are groupable by individual status code (200, 403, 404) or by category (2xx, 3xx, 4xx, 5xx).

The same widget is available on each verified bot’s detail page (only available for AI bots), for example Google.

Explore all three features on the Cloudflare Radar AI Insights page.

Cloudflare’s network now supports redirecting verified AI training crawlers to canonical URLs when they request deprecated or duplicate pages. When enabled via AI Crawl Control > Quick Actions, AI training crawlers that request a page with a canonical tag pointing elsewhere receive a 301 redirect to the canonical version. Humans, search engine crawlers, and AI Search agents continue to see the original page normally.

This feature leverages your existing <link rel="canonical"> tags. No additional configuration required beyond enabling the toggle. Available on Pro, Business, and Enterprise plans at no additional cost.

Refer to the Redirects for AI Training documentation for details.

New AI Search instances created after today will work differently. New instances come with built-in storage and a vector index, so you can upload a file, have it indexed immediately, and search it right away.

Additionally new Workers Bindings are now available to use with AI Search. The new namespace binding lets you create and manage instances at runtime, and cross-instance search API lets you query across multiple instances in one call.

Built-in storage and vector index

All new instances now comes with built-in storage which allows you to upload files directly to it using the Items API or the dashboard. No R2 buckets to set up, no external data sources to connect first.

const instance = env.AI_SEARCH.get("my-instance");

// upload and wait for indexing to complete
const item = await instance.items.uploadAndPoll("faq.md", content);

// search immediately after indexing
const results = await instance.search({
  messages: [{ role: "user", content: "onboarding guide" }],
});

Namespace binding

The new ai_search_namespaces binding replaces the previous env.AI.autorag() API provided through the AI binding. It gives your Worker access to all instances within a namespace and lets you create, update, and delete instances at runtime without redeploying.

// wrangler.jsonc
{
  "ai_search_namespaces": [
    {
      "binding": "AI_SEARCH",
      "namespace": "default",
    },
  ],
}

// create an instance at runtime
const instance = await env.AI_SEARCH.create({
  id: "my-instance",
});

For migration details, refer to Workers binding migration. For more on namespaces, refer to Namespaces.

Cross-instance search

Within the new AI Search binding, you now have access to a Search and Chat API on the namespace level. Pass an array of instance IDs and get one ranked list of results back.

const results = await env.AI_SEARCH.search({
  messages: [{ role: "user", content: "What is Cloudflare?" }],
  ai_search_options: {
    instance_ids: ["product-docs", "customer-abc123"],
  },
});

Refer to Namespace-level search for details.