agentskillexchange Scrapy Pipeline Data Extractor
Builds production Scrapy spiders with custom Item Pipelines for data cleaning and storage. Uses scrapy.linkextractors.LinkExtractor for crawl scoping and ItemLoader with MapCompose processors for field normalization.
Installation
Use the upstream install or setup path that matches your environment:
- pip install scrapy
Requirements and caveats from upstream:
- :alt: Supported Python Versions
- It is cross-platform, and requires Python 3.10+. It is maintained by Zyte_
Basic usage or getting-started notes:
.. code:: bash
And follow the documentation_ to learn how to use it.
.. _documentation: https://docs.scrapy.org/en/latest/
Source: https://github.com/scrapy/scrapy
Extracted from upstream docs: https://raw.githubusercontent.com/scrapy/scrapy/HEAD/README.rst