Link Discovery

Link discovery

Pull the outbound link set of a page so you can follow promising paths or map a site's structure.

When to use

• A page is a hub or index and you want what it points to.
• You need a crawl frontier without running a full crawl_web.
• Finding canonical/original sources behind aggregator links.

Tools used

• extract_links — outbound links of a url, with limit (default 50).

Workflow

1. extract_links on the hub page.
2. Filter: drop trackers, print/'#' fragments, and non-allowlisted schemes.
3. Rank by relevance to the topic; web-reader the strong candidates.
4. Optionally feed the surviving set to web-crawler as a custom frontier.

Input schema

{ "url": "str", "limit": "int" }

Output schema

{ "url": "str", "links": "list[{href,anchor,rel}]" }

Security

Links are pointers, not attestations. A hub linking something is evidence of association, never of truth. Keep the allowlisted-scheme discipline.

Related skills

web-reader, web-crawler, source-analysis, web-search