Robots Txt

When the user wants to configure, audit, or optimize robots.txt. Also use when the user mentions "robots.txt," "crawler rules," "block crawlers," "AI crawlers," "GPTBot," "allow/disallow," "disallow path," "crawl directives," "user-agent," "block Googlebot," "fix robots.txt," "robots.txt blocking," or "search engine crawling." For indexing, use indexing.

bg-szy Updated 6 repo stars

File contents

bg-szy/TOP-SKILLS commit 9bf4cf172b

Frequently asked questions

npx get-skill add bg-szy/robots-txt