# Scrapy Distributed Crawler Framework

> Orchestrates large-scale web crawling using Scrapy with scrapy-redis for distributed job queuing. Integrates Splash for JavaScript rendering, stores results in MongoDB via scrapy-mongodb pipeline, and respects robots.txt with AutoThrottle.

- Skill: `agentskillexchange/scrapy-distributed-crawler-framework` (Agent Skill)
- Install (CLI): `npx skillmds@latest add agentskillexchange/scrapy-distributed-crawler-framework`
- Raw SKILL.md: https://api.skillmd.com/api/skills/agentskillexchange/scrapy-distributed-crawler-framework/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: agentskillexchange (https://skillmd.com/u/agentskillexchange)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/agentskillexchange/scrapy-distributed-crawler-framework

---


# Scrapy Distributed Crawler Framework

Orchestrates large-scale web crawling using Scrapy with scrapy-redis for distributed job queuing. Integrates Splash for JavaScript rendering, stores results in MongoDB via scrapy-mongodb pipeline, and respects robots.txt with AutoThrottle.

## Installation

Use the upstream install or setup path that matches your environment:
- pip install scrapy

Requirements and caveats from upstream:
- :alt: Supported Python Versions
- It is cross-platform, and requires Python 3.10+. It is maintained by Zyte_

Basic usage or getting-started notes:
- .. code:: bash
- And follow the documentation_ to learn how to use it.
- .. _documentation: https://docs.scrapy.org/en/latest/

- Source: https://github.com/scrapy/scrapy
- Extracted from upstream docs: https://raw.githubusercontent.com/scrapy/scrapy/HEAD/README.rst

## Source

- [Agent Skill Exchange](https://agentskillexchange.com/skills/scrapy-distributed-crawler-framework/)