{"id":385,"date":"2026-04-17T00:00:00","date_gmt":"2026-04-17T00:00:00","guid":{"rendered":"https:\/\/wordpress.securinsight.ca\/index.php\/2026\/04\/17\/ai-crawl-control-tools-to-prepare-your-site-for-the-agentic-internet-2\/"},"modified":"2026-04-17T00:00:00","modified_gmt":"2026-04-17T00:00:00","slug":"ai-crawl-control-tools-to-prepare-your-site-for-the-agentic-internet-2","status":"publish","type":"post","link":"https:\/\/wordpress.securinsight.ca\/index.php\/2026\/04\/17\/ai-crawl-control-tools-to-prepare-your-site-for-the-agentic-internet-2\/","title":{"rendered":"AI Crawl Control &#8211; Tools to prepare your site for the agentic Internet"},"content":{"rendered":"<p>AI Crawl Control now includes new tools to help you prepare your site for the agentic Internet\u2014a web where AI agents are first-class citizens that discover and interact with content differently than human visitors.<\/p>\n<h4>Content Format insights<\/h4>\n<p>The <strong>Metrics<\/strong> tab now includes a <strong>Content Format<\/strong> chart showing what content types AI systems request versus what your origin serves. Understanding these patterns helps you optimize content delivery for both human and agent consumption.<\/p>\n<h4>Directives tab (formerly Robots.txt)<\/h4>\n<p>The <strong>Robots.txt<\/strong> tab has been renamed to <strong>Directives<\/strong> and now includes a link to check your site&#8217;s <a href=\"https:\/\/isitagentready.com\/\" target=\"_blank\">Agent Readiness<\/a> score.<\/p>\n<p>Refer to our <a href=\"https:\/\/blog.cloudflare.com\/agent-readiness\/\" target=\"_blank\">blog post on preparing for the agentic Internet<\/a> for more on why these capabilities matter.<\/p>","protected":false},"excerpt":{"rendered":"<p>AI Crawl Control now includes new tools to help you prepare your site for the agentic Internet\u2014a web where AI agents are first-class citizens that discover and interact with content differently than human visitors. Content Format insights The Metrics tab now includes a Content Format chart showing what content types AI systems request versus what [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-385","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/wordpress.securinsight.ca\/index.php\/wp-json\/wp\/v2\/posts\/385","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/wordpress.securinsight.ca\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/wordpress.securinsight.ca\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/wordpress.securinsight.ca\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/wordpress.securinsight.ca\/index.php\/wp-json\/wp\/v2\/comments?post=385"}],"version-history":[{"count":0,"href":"https:\/\/wordpress.securinsight.ca\/index.php\/wp-json\/wp\/v2\/posts\/385\/revisions"}],"wp:attachment":[{"href":"https:\/\/wordpress.securinsight.ca\/index.php\/wp-json\/wp\/v2\/media?parent=385"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/wordpress.securinsight.ca\/index.php\/wp-json\/wp\/v2\/categories?post=385"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/wordpress.securinsight.ca\/index.php\/wp-json\/wp\/v2\/tags?post=385"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}