Results for “common-crawl”

Bulk extract content from an entire website or site section by crawling pages that follow links, with configurable depth, path filters, and concurrency.

Search the web, scrape pages, crawl sites, and interact with dynamic content via the Firecrawl CLI, returning clean markdown for LLM contexts.

Deep-crawl any website from start URLs, returning per-page LLM-ready text, markdown, or HTML with metadata and in-scope outbound links.

Automate web crawling and data extraction with Firecrawl: scrape pages, crawl sites, extract structured data, batch scrape URLs, and map website structures.

Detect adversary lateral movement across networks using Splunk SPL queries against Windows authentication logs, SMB traffic, and remote service abuse.

Discovers and lists all URLs on a website, with optional search filtering to find specific pages within large sites.

Extracts clean, LLM-optimized markdown from any URL, including JavaScript-rendered SPAs, with support for concurrent scraping of multiple URLs and options like main-content-only extraction and custom output formats.