Results for “url-discovery”

Deep-crawl any website from start URLs, returning per-page LLM-ready text, markdown, or HTML with metadata and in-scope outbound links.

Crawl a live website, extract its design system, brand surface, and page inventory, and save the snapshot under stardust/current/ for downstream redesign or migration.

Search, browse, and analyze content on Xiaohongshu (Little Red Book) including feeds, note details, comments, and user profiles.

Search the web and fetch content from URLs, returning LLM-friendly markdown.

Searches the web for text, news, images, videos, and books using the duckse CLI, returning clean results in text or JSON format.

Performs web search, content extraction, site crawling, URL discovery, and deep research via the Tavily CLI, returning JSON for LLM consumption.

Scans a codebase for broken links and references, auto-fixes what it can, and flags what needs human review.

Bulk extract content from an entire website or site section by crawling pages that follow links, with configurable depth, path filters, and concurrency.

Crawls websites and saves content from multiple pages as local markdown files using the Tavily CLI, with options for depth, breadth, path filtering, and semantic focus.

Searches the web via the Browserbase Search API and returns structured results with titles, URLs, authors, and dates, without needing a full browser session.

Searches the web and optionally extracts full page content, returning results as JSON files.

Searches the web for text, news, images, videos, and books using the duckse CLI, with options for time filters, regions, safe search, and JSON output.

Fetches any URL via Chrome CDP and converts HTML to clean markdown, supporting auto-capture and wait-for-user modes for login-required pages.

Searches for existing content pages containing a specific block in AEM Edge Delivery Services, reporting URLs with occurrences and variants to help identify test content during development.