Agent Ready Robots Txt

Sub-skill de agent-ready-cloudflare: Implement robots.txt

fabricioctelles Updated

File contents

Implement robots.txt

Publish a valid robots.txt at your site root per RFC 9309.

Requirements

  • Serve /robots.txt as text/plain with HTTP 200
  • Include User-agent directives with Allow/Disallow rules
  • Reference your sitemap if one exists: Sitemap: https://example.com/sitemap.xml

Cloudflare

AI Crawl Control can manage your robots.txt from the dashboard, including AI-specific bot rules.

Validate

POST https://isitagentready.com/api/scan
Content-Type: application/json

{"url": "https://YOUR-SITE.com"}

Check that checks.discoverability.robotsTxt.status is "pass".

fabricioctelles/skills/tree/main/skills/agent-ready-cloudflare/robots-txt commit 0b18c421b7

Frequently asked questions

npx skillmds@latest add fabricioctelles/agent-ready-robots-txt