Context Scrapers

Web scraping spiders (Scrapy), parsers, and source management.

majiayu000 e0ba5ca 2 files · 1.6 KB Updated 567 repo stars

File contents

Scrapers Context

Overview

Web scraping infrastructure using Scrapy. Handles data acquisition from city councils (Legistar) and municipal codes (Municode).

Active Files

Spiders

  • backend/affordabot_scraper/affordabot_scraper/spiders/sanjose_meetings.py - Legistar meeting scraper
  • backend/affordabot_scraper/affordabot_scraper/spiders/sanjose_municode.py - Municode scraper

Configuration

  • backend/affordabot_scraper/affordabot_scraper/pipelines.py - Item pipelines (DB storage)
  • backend/affordabot_scraper/affordabot_scraper/settings.py - Scrapy settings

Verification Scripts

  • scripts/verify_raw_scrapes.py - Check raw_scrapes table
  • scripts/verify_municode_discovery.py - Verify discovery logic

Usage

Use this skill when modifying scrapers, adding new jurisdictions, or debugging ingestion.

majiayu000/claude-skill-registry-data/tree/main/data/context-scrapers commit e0ba5cad30

Frequently asked questions

npx skillmds add majiayu000/context-scrapers