# Robots Txt

> Use when applies to any public website. Use when auditing technical SEO foundations or diagnosing crawl coverage gaps.

- Skill: `thedaviddias/robots-txt` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add thedaviddias/robots-txt`
- Raw SKILL.md: https://api.skillmd.com/api/skills/thedaviddias/robots-txt/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Marketing & Growth
- Author: thedaviddias (https://skillmd.com/u/thedaviddias)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/thedaviddias/robots-txt

---


# Publish a robots.txt file

robots.txt is the first file crawlers fetch; misconfigured directives can silently block search engines from crawling your entire site, killing organic visibility.

## Quick Reference

- Serve a valid `robots.txt` at `/robots.txt` on the production domain, returning HTTP 200
- Include a `Sitemap:` directive pointing to your XML sitemap
- Never disallow crawling of CSS/JS assets that render your pages
- Avoid blocking all crawlers with `Disallow: /` on a live site

## Check

Fetch `/robots.txt` on the live domain and verify it returns HTTP 200, uses correct `User-agent` / `Disallow` / `Allow` syntax, and includes a `Sitemap:` directive pointing to the XML sitemap. Check for accidental `Disallow: /` directives.

## Fix

Create or update `robots.txt` at the web root with valid directives. Add a live `Sitemap:` line for the production sitemap URL. Remove any `Disallow: /` rules that block the whole site or resources needed for rendering.

## Explain

Explain how robots.txt controls crawler access, why an accidental `Disallow: /` can delist a site, and why CSS/JS must remain accessible for rendering-based indexing.

## Code Review

Review metadata generation, rendered HTML, structured data, and response headers related to Publish a robots.txt file. Flag exact routes or templates where search-facing output violates the rule, and describe how to verify the final page output.

---

For full implementation details, code examples, and framework-specific guidance,
see `references/rule.md`.

Rule page: https://frontendchecklist.io/en/rules/seo/robots-txt