Web SEO Crawl Indexing

Use before changing robots.txt or deciding what crawlers may fetch - pages that stay in search results after being blocked, crawler load you want to cut, an asset or image host you are tempted to block, a URL space multiplied by language or filter parameters, or a stale sitemap. Keywords - robots.txt, Disallow, Allow, noindex, deindex, crawl budget, crawl trap, Googlebot, image search, Merchant Center, sitemap.

bitranox Updated 1 repo stars

File contents

bitranox/bitranox-skills commit a14fbb129d

Frequently asked questions

npx get-skill add bitranox/web-seo-crawl-indexing