Agent Ready AI Rules

Sub-skill de agent-ready-cloudflare: Implement AI Bot Rules in robots.txt

fabricioctelles Updated

File contents

Implement AI Bot Rules in robots.txt

Add explicit User-agent entries for AI crawlers in your robots.txt per RFC 9309.

Requirements

  • Add User-agent blocks for AI-specific bots: GPTBot, OAI-SearchBot, Claude-Web, Google-Extended, Amazonbot, anthropic-ai, Bytespider, CCBot, Applebot-Extended
  • Set Allow and/or Disallow rules matching your content policy
  • A wildcard User-agent: * block alone is not sufficient — explicit AI bot entries are required

Cloudflare

AI Crawl Control lets you manage AI bot rules from the dashboard without editing robots.txt manually.

Validate

POST https://isitagentready.com/api/scan
Content-Type: application/json

{"url": "https://YOUR-SITE.com"}

Check that checks.botAccessControl.robotsTxtAiRules.status is "pass".

fabricioctelles/skills/tree/main/skills/agent-ready-cloudflare/ai-rules commit c3ec65bcf3

Frequently asked questions

npx skillmds@latest add fabricioctelles/agent-ready-ai-rules