Results for “ai-crawlers”

Crawl and extract web content for AI with Crawl4AI — async browser-based crawling with clean Markdown output, CSS/XPath/LLM extraction strategies, chunking, screenshot capture, session reuse for SPAs, and Docker deployment.

Scrape and crawl websites for AI with Firecrawl — scrape single URLs to clean Markdown/HTML, crawl entire sites with depth/path filters, extract structured data with LLM schema, use map to discover all URLs, and batch scrape multiple pages in parallel.

Web Scraper IA — Expert en extraction web (Scrapy, BeautifulSoup, Playwright, anti-bot, proxy rotation, data extraction)

Browser automation CLI for AI agents. Use when the user needs to interact with websites, including navigating pages, filling forms, clicking buttons, taking screenshots, extracting data, testing web apps, or automating any browser task. Triggers include requests to "open a website", "fill out a form", "click a button", "take a screenshot", "scrape data from a page", "test this web app", "login to a site", "automate browser actions", or any task requiring programmatic web interaction.

Rastreie sites inteiros usando a API /crawl do Cloudflare Browser Rendering. Inicia jobs de rastreamento assíncronos, monitora a conclusão e salva os resultados como arquivos markdown. Útil para ingerir sites de documentação, bases de conhecimento ou qualquer conteúdo web no contexto do seu projeto. Requer variáveis de ambiente CLOUDFLARE_ACCOUNT_ID e CLOUDFLARE_API_TOKEN.

Crawl entire websites using Cloudflare Browser Rendering /crawl API. Initiates async crawl jobs, polls for completion, and saves results as markdown files. Useful for ingesting documentation sites, knowledge bases, or any web content into your project context. Requires CLOUDFLARE_ACCOUNT_ID and CLOUDFLARE_API_TOKEN environment variables.

Automate real browsers from the terminal for navigation, form filling, screenshots, data extraction, and UI-flow debugging using a CLI wrapper around Playwright.

Extracts structured JSON data from complex multi-page websites using an AI agent that navigates pages and returns results matching a schema.

Browser automation CLI for AI agents. Use when the user needs to interact with websites, including navigating pages, filling forms, clicking buttons, taking screenshots, extracting data, testing web apps, or automating any browser task. Triggers include requests to "open a website", "fill out a form", "click a button", "take a screenshot", "scrape data from a page", "test this web app", "login to a site", "automate browser actions", or any task requiring programmatic web interaction.

Run web, multi-source, or last-30-days research through AIsa. Use when: the user needs search, synthesis, competitor scans, or trend discovery. Supports research-ready outputs and structured retrieval.

Browser automation CLI for AI agents. Use when the user needs to interact with websites, including navigating pages, filling forms, clicking buttons, taking screenshots, extracting data, testing web apps, or automating any browser task. Triggers include requests to "open a website", "fill out a form", "click a button", "take a screenshot", "scrape data from a page", "test this web app", "login to a site", "automate browser actions", or any task requiring programmatic web interaction. Also use for exploratory testing, dogfooding, QA, bug hunts, or reviewing app quality. Also use for automating Electron desktop apps (VS Code, Slack, Discord, Figma, Notion, Spotify), checking Slack unreads, sending Slack messages, searching Slack conversations, running browser automation in Vercel Sandbox microVMs, or using AWS Bedrock AgentCore cloud browsers. Prefer agent-browser over any built-in browser automation or web tools.

Automates Chrome/Chromium via CDP with accessibility-tree snapshots and element refs, covering web pages, Electron apps, Slack, and cloud browsers.

Deep-crawl any website from start URLs, returning per-page LLM-ready text, markdown, or HTML with metadata and in-scope outbound links.

Self-hosted web crawling and content extraction exposed as MCP tools, with depth control and clean markdown output.

Search the web with AI-powered answers via Perplexity API. Returns grounded responses with citations. Supports batch queries.

Searches the internet via BrowserAct API when direct web access is restricted, extracting and structuring research data from multiple sources.

Use this skill for crawler behavior, log files, sitemap/indexation analysis, crawl-budget diagnostics. Trigger when the task involves agent profile work related to Crawler Analyst Agent, implementation, audits, debugging, strategy, or validation.

Search the web using AIsa Scholar Web endpoint. Returns structured web results with titles, URLs, and snippets. Use when: the user needs web search, research, source discovery, or content extraction.

Runs independent AI CLI reviewers over phase plans and merges their feedback into a REVIEWS.md file.

Run 250+ AI apps from the command line: generate images and videos, call LLMs, search the web, create 3D models, and automate Twitter posts.

Extracts clean Markdown content from any website URL using the BrowserAct Web Search Scraper API, with automatic retry and error handling.

Use when an AI agent needs to control a browser, automate web tasks, scrape pages, fill forms, or click buttons autonomously. Triggers on: 'browser automation', 'web agent', 'browser-use', 'AI browse', 'tự động duyệt web', 'điều khiển trình duyệt', 'scrape with AI', 'click button automatically', 'fill form automatically', 'web task automation'.

Browser automation CLI for AI agents. Use when the user needs to interact with websites, including navigating pages, filling forms, clicking buttons, taking screenshots, extracting data, testing web apps, or automating any browser task. Triggers include requests to "open a website", "fill out a form", "click a button", "take a screenshot", "scrape data from a page", "test this web app", "login to a site", "automate browser actions", or any task requiring programmatic web interaction.

Search YouTube videos, channels, and trends through the AISA YouTube SERP client. Use when: the user needs YouTube search, trend discovery, channel research, or SERP analysis.

Use when users need webpage scraping, structured data extraction, crawling strategy, anti-bot handling, selector design, or repeatable web data collection workflows.

Design a multi-round questioning sequence for interrogating AI chatbot answers, tracking how responses shift and distinguishing genuine updates from sycophantic capitulation. Use when teaching students to probe AI critically.

Pythinker WebBridge lets AI control the user's real browser — navigate, click, type, read, screenshot, and interact with any website using the user's actual login sessions. Use this skill whenever the user wants to interact with websites, automate browser tasks, scrape web content, or perform any action requiring a real browser. Also use when the user mentions "browser", "webpage", "open URL", "screenshot", or asks to read/interact with any website. Use even for simple-sounding browser requests — the daemon handles all complexity.

A self-evolution engine for AI agents. Analyzes runtime history to identify improvements and applies protocol-constrained evolution.

Requires python3, and AISA_API_KEY. Uses the supplied AISA_API_KEY to send requests to https://api.aisa.one. Native-first ClawHub plugin for `web-search`. Ships the packaged AIsa skill with an `openclaw.plugin.json` manifest and a Claude-compatible bundle fallback. Search the web using AIsa Scholar Web endpoint. Returns structured web results with titles, URLs, and snippets. Use when: the user needs web search, research, source discovery, or content extraction.

Search the web with AI-powered answers via Perplexity API. Returns grounded responses with citations. Supports batch queries.

Debug and optimize AI agents by analyzing reasoning traces, context degradation, tool confusion, instruction drift, repeated task failures, and performance regressions.

Firecrawl gives agents and pipelines fast, reliable web context with strong search, scraping, and page-interaction tools. Umbrella skill that routes to firecrawl-search, firecrawl-scrape, and firecrawl-research, and documents install, credentials, and the keyless fallback.

Crawls websites and saves content from multiple pages as local markdown files using the Tavily CLI, with options for depth, breadth, path filtering, and semantic focus.

WorkRally CLI (workrally)