Scrapy Pipeline Manager
Manages Scrapy spider deployments via Scrapyd API with custom item pipelines for MongoDB ingestion, deduplication via MinHash LSH, and rotating proxy middleware configuration.
Installation
Use the upstream install or setup path that matches your environment:
- pip install scrapy
Requirements and caveats from upstream:
- :alt: Supported Python Versions
- It is cross-platform, and requires Python 3.10+. It is maintained by Zyte_
Basic usage or getting-started notes:
.. code:: bash
And follow the documentation_ to learn how to use it.
.. _documentation: https://docs.scrapy.org/en/latest/
Source: https://github.com/scrapy/scrapy
Extracted from upstream docs: https://raw.githubusercontent.com/scrapy/scrapy/HEAD/README.rst