Scrapy Pipeline Manager

Manages Scrapy spider deployments via Scrapyd API with custom item pipelines for MongoDB ingestion, deduplication via MinHash LSH, and rotating proxy middleware configuration.

agentskillexchange Updated 28 repo stars

File contents

Scrapy Pipeline Manager

Manages Scrapy spider deployments via Scrapyd API with custom item pipelines for MongoDB ingestion, deduplication via MinHash LSH, and rotating proxy middleware configuration.

Installation

Use the upstream install or setup path that matches your environment:

  • pip install scrapy

Requirements and caveats from upstream:

  • :alt: Supported Python Versions
  • It is cross-platform, and requires Python 3.10+. It is maintained by Zyte_

Basic usage or getting-started notes:

Source

agentskillexchange/skills/tree/main/skills/scrapy-pipeline-manager commit bc9f75b701

Frequently asked questions

npx skillmds@latest add agentskillexchange/scrapy-pipeline-manager