Results for “url-discovery”

Deep-crawl any website from start URLs, returning per-page LLM-ready text, markdown, or HTML with metadata and in-scope outbound links.

Search, browse, and analyze content on Xiaohongshu (Little Red Book) including feeds, note details, comments, and user profiles.

Search the web, extract URL content, enrich datasets with web-sourced fields, and run deep research reports, prioritizing academic and scientific sources.

Bulk extract content from an entire website or site section by crawling pages that follow links, with configurable depth, path filters, and concurrency.

Search the web and extract content from URLs using Exa, with support for academic and scientific sources.

Detects DNS tunneling by computing Shannon entropy of DNS query names, analyzing query length distributions, inspecting TXT record payloads, and identifying high subdomain cardinality using scapy for packet capture analysis.

Extracts clean, LLM-optimized markdown from any URL, including JavaScript-rendered SPAs, with support for concurrent scraping of multiple URLs and options like main-content-only extraction and custom output formats.

Generate an OpenAPI 3.1 specification from a browser-trace capture by analyzing observed HTTP traffic, templating URLs, and inferring JSON schemas from request/response samples.