# Agent Ready Robots Txt

> Sub-skill de agent-ready-cloudflare: Implement robots.txt

- Skill: `fabricioctelles/agent-ready-robots-txt` (Agent Skill)
- Install (CLI): `npx skillmds@latest add fabricioctelles/agent-ready-robots-txt`
- Raw SKILL.md: https://api.skillmd.com/api/skills/fabricioctelles/agent-ready-robots-txt/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: fabricioctelles (https://skillmd.com/u/fabricioctelles)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/fabricioctelles/agent-ready-robots-txt

---

# Implement robots.txt

Publish a valid robots.txt at your site root per
[RFC 9309](https://www.rfc-editor.org/rfc/rfc9309).

## Requirements

- Serve `/robots.txt` as `text/plain` with HTTP 200
- Include `User-agent` directives with `Allow`/`Disallow` rules
- Reference your sitemap if one exists: `Sitemap: https://example.com/sitemap.xml`

## Cloudflare

[AI Crawl Control](https://developers.cloudflare.com/ai-crawl-control/)
can manage your robots.txt from the dashboard, including AI-specific bot rules.

## Validate

```
POST https://isitagentready.com/api/scan
Content-Type: application/json

{"url": "https://YOUR-SITE.com"}
```

Check that `checks.discoverability.robotsTxt.status` is `"pass"`.