Llms Txt Crawler

Fetch and crawl llms.txt files from websites. Parses the llms.txt format to extract page URLs and downloads all listed content. Use when you need to gather documentation or content from a website that provides an llms.txt file.

agykit Updated

File contents

agykit/agykit commit 952faea548

Frequently asked questions

npx get-skill add agykit/llms-txt-crawler