Scraping Data Pipeline

You are an expert at orchestrating the UFC Pokedex scraping pipeline, which involves multiple steps from data collection to database loading.

Pipeline Overview

The scraping pipeline follows this flow:

UFCStats.com → Scrapy Spiders → JSON/JSONL files → Validation → Database → Cache Invalidation

When to Use This Skill

Invoke this skill when the user wants to:

• Scrape fighter data (list or details)
• Load scraped data into the database
• Validate scraped JSON/JSONL files
• Run the complete scraping pipeline
• Handle missing or failed scrapes
• Update fighter data from UFC Stats

Available Operations

1. Scraping Operations

Scrape Fighter List

Scrapes all fighter URLs from UFCStats.com alphabetical listing.

Command:

make scraper

Output: data/processed/fighters_list.jsonl

What it does:

• Crawls http://ufcstats.com/statistics/fighters alphabetically
• Extracts fighter IDs and basic info
• Saves to JSONL format (one fighter per line)
• Takes ~5-10 minutes for full list

Scrape Fighter Details

Scrapes detailed information for specific fighters.

Command options:

# Sample scrape (single fighter for testing)
make scrape-sample

# All fighters from list
make scraper-details

# Specific fighters by ID
.venv/bin/scrapy crawl fighter_detail -a fighter_ids=id1,id2,id3

# Specific fighters by URL
.venv/bin/scrapy crawl fighter_detail -a fighter_urls=url1,url2

Output: data/processed/fighters/{id}.json (individual files)

What it scrapes:

• Personal info (name, nickname, DOB, height, weight, reach, stance)
• Complete fight history with results
• Career statistics
• Current record (W-L-D)

2. Data Loading Operations

Load Sample Data (SQLite-safe)

Loads first 8 fighters from fixtures for quick testing.

Command:

make api:seed

Use when:

• Testing on SQLite
• Need quick test data
• Developing new features

Load All Scraped Data

Loads complete scraped dataset into database.

Command:

make load-data

Use when:

• Have full scraped dataset
• Running PostgreSQL
• Need production data

Safety note: Blocked on SQLite by default (10K+ fighters). Override with:

ALLOW_SQLITE_PROD_SEED=1 make api:seed-full

Load Sample of Scraped Data

Loads only first N fighters for testing.

Command:

make load-data-sample

Use when:

• Testing with realistic data
• Need more than 8 fighters but not full dataset
• Validating data loading logic

3. Data Validation

Before loading, validate the scraped data:

Steps:

1. Check files exist:

   ls -lh data/processed/fighters_list.jsonl
   ls -lh data/processed/fighters/*.json | wc -l
   

2. Validate JSONL structure:

   head -5 data/processed/fighters_list.jsonl | jq '.'
   

3. Check for parsing errors:

   grep -i "error\|exception" data/processed/fighters_list.jsonl
   

4. Count records:

   wc -l data/processed/fighters_list.jsonl
   

Complete Pipeline Workflows

Workflow 1: Full Scrape and Load (PostgreSQL)

Use this for complete data refresh on PostgreSQL.

Steps:

# 1. Ensure PostgreSQL is running
docker-compose up -d

# 2. Scrape fighter list (~5-10 min)
make scraper

# 3. Validate fighter list
wc -l data/processed/fighters_list.jsonl
head -3 data/processed/fighters_list.jsonl | jq '.'

# 4. Scrape fighter details (~30-60 min for all)
make scraper-details

# 5. Validate detail files
ls data/processed/fighters/*.json | wc -l

# 6. Load into database
make load-data

# 7. Verify in database
PGPASSWORD=ufc_pokedex psql -h localhost -p 5432 -U ufc_pokedex -d ufc_pokedex -c "SELECT COUNT(*) FROM fighters;"

# 8. Restart backend to invalidate cache
pkill -f uvicorn
make api

Expected duration: 40-90 minutes total

Workflow 2: Sample Scrape and Load (Quick Test)

Use this for quick testing or development.

Steps:

# 1. Start with SQLite (no Docker needed)
USE_SQLITE=1

# 2. Scrape sample fighter
make scrape-sample

# 3. Seed with fixtures
make api:seed

# 4. Start backend
make api:dev

# 5. Verify
curl http://localhost:8000/fighters/ | jq '.fighters | length'

Expected duration: 2-3 minutes total

Workflow 3: Update Specific Fighters

Use this to refresh data for specific fighters.

Steps:

# 1. Get fighter IDs that need updating
# (from database or fighters_list.jsonl)

# 2. Scrape specific fighters
.venv/bin/scrapy crawl fighter_detail -a fighter_ids=id1,id2,id3

# 3. Reload just those fighters
# (Currently requires full reload - see limitations below)
make load-data

# 4. Restart backend
pkill -f uvicorn && make api

Workflow 4: Handle Missing/Failed Scrapes

If some fighters failed to scrape:

Steps:

# 1. Find fighters with missing detail files
# Compare fighters_list.jsonl count vs. fighters/*.json count
TOTAL=$(wc -l < data/processed/fighters_list.jsonl)
SCRAPED=$(ls data/processed/fighters/*.json 2>/dev/null | wc -l)
echo "Total: $TOTAL, Scraped: $SCRAPED, Missing: $((TOTAL - SCRAPED))"

# 2. Identify missing IDs
# (You may need to write a quick script)

# 3. Re-scrape missing fighters
.venv/bin/scrapy crawl fighter_detail -a fighter_ids=missing_id1,missing_id2

# 4. Reload data
make load-data

Cache Management

After loading data, invalidate Redis cache (if using Redis):

Commands:

# Check Redis connection
redis-cli ping

# Clear all fighter-related cache
redis-cli KEYS "fighters:*" | xargs redis-cli DEL

# Or clear entire cache
redis-cli FLUSHDB

Note: Backend gracefully degrades if Redis is unavailable, so cache clearing is optional.

Error Handling

Common Issues and Solutions

Issue: "No such file or directory: data/processed/fighters_list.jsonl"

Solution: Run make scraper first to create the fighter list.

Issue: Scraper fails with connection errors

Solution:

• Check internet connection
• Verify UFCStats.com is accessible
• Reduce concurrent requests in scraper/settings.py
• Add delays: SCRAPER_DELAY_SECONDS=2.0

Issue: Database constraint violations during load

Solution:

• Drop and recreate database: make db-reset
• Check for duplicate IDs in scraped data
• Validate JSON structure

Issue: "Blocked on SQLite" when loading full dataset

Solution:

• Use PostgreSQL for production data: docker-compose up -d
• Or override safety check: ALLOW_SQLITE_PROD_SEED=1 make api:seed-full

Issue: Partial scrapes (some fighters missing)

Solution:

• Use Workflow 4 above to identify and re-scrape missing fighters
• Check scraper logs for errors
• Verify HTML structure hasn't changed on UFCStats.com

Data Validation Checklist

Before loading, verify:

• fighters_list.jsonl exists and has expected count (~1000-2000 fighters)
• Each line in JSONL is valid JSON
• Fighter detail JSON files exist in data/processed/fighters/
• Sample fighter JSON has required fields: id, name, record
• No parsing errors in JSON files
• Database is running (PostgreSQL) or SQLite mode enabled
• Database schema is up to date: make db-upgrade

Progress Monitoring

Monitor scraping progress:

# Watch fighter details being created
watch -n 5 'ls data/processed/fighters/*.json | wc -l'

# Check scrapy logs
tail -f scrapy.log

Monitor database loading:

# Check fighter count in database
PGPASSWORD=ufc_pokedex psql -h localhost -p 5432 -U ufc_pokedex -d ufc_pokedex -c "SELECT COUNT(*) FROM fighters;"

# Check recent inserts
PGPASSWORD=ufc_pokedex psql -h localhost -p 5432 -U ufc_pokedex -d ufc_pokedex -c "SELECT name, record, created_at FROM fighters ORDER BY created_at DESC LIMIT 10;"

Best Practices

1. Always validate before loading - Check JSONL structure and counts
2. Use sample loads for testing - Don't load 10K fighters into SQLite
3. Monitor scraping progress - Scrapy can fail silently on some fighters
4. Respect rate limits - UFCStats.com should not be hammered
5. Clear cache after loading - Ensure API serves fresh data
6. Back up before full reload - Database drops and recreates tables
7. Test with sample first - Validate pipeline with make scrape-sample

Limitations

• Incremental updates not supported - Must reload all data (no upsert logic yet)
• No automatic scheduling - Must trigger scrapes manually
• No fighter image scraping - Images handled separately (see managing-fighter-images skill)
• SQLite safety checks - Full dataset blocked on SQLite (design choice)
• No event scraping yet - Events spider exists but not integrated

Quick Reference

# Full pipeline (PostgreSQL)
make scraper && make scraper-details && make load-data

# Quick test (SQLite)
make scrape-sample && make api:seed && make api:dev

# Validate data
wc -l data/processed/fighters_list.jsonl
ls data/processed/fighters/*.json | wc -l

# Clear cache
redis-cli FLUSHDB

# Check database
PGPASSWORD=ufc_pokedex psql -h localhost -U ufc_pokedex -d ufc_pokedex -c "SELECT COUNT(*) FROM fighters;"