Normalize and filter noisy URL lists before crawling or queueing

Uses Courlan to clean, normalize, de-track, and language-filter raw URL inventories before a crawler, scraper, or analyst queue touches them. Best when an agent already has too many candidate links and needs a smaller, cleaner frontier, not a full crawling stack.

agentskillexchange Updated 28 repo stars

File contents

agentskillexchange/skills/tree/main/skills/normalize-and-filter-noisy-url-lists-before-crawling-or-queueing commit f70dde3700

Frequently asked questions

npx skillmds@latest add agentskillexchange/normalize-and-filter-noisy-url-lists-before-crawling-or-queu