X Twitter Scraper

Twitter/X Scraper

Name: twitter-scraper
Version: 1.0.0
Runtime: python3
Browser: chromium (via Playwright)

Overview

Two-phase scraping system for Twitter/X public profiles:

1. Profile Discovery — Find Twitter accounts via Google Custom Search API or DuckDuckGo (searches x.com and twitter.com)
2. Browser Scraping — Scrape public profiles using Playwright with anti-detection (no login required)

Setup

# Install Python dependencies
pip install -r requirements.txt

# Install Playwright browser
playwright install chromium

Google Custom Search API (Optional)

For discovery via Google:

1. Create API key at Google Cloud Console
2. Create Programmable Search Engine at cse.google.com scoped to x.com and twitter.com
3. Set values in config/scraper_config.json:

   {
     "google_search": {
       "enabled": true,
       "api_key": "YOUR_API_KEY",
       "search_engine_id": "YOUR_CX_ID"
     }
   }
   

If not configured, discovery falls back to DuckDuckGo (no API key needed).

Commands

Discover Profiles

# Interactive mode
python main.py discover

# Specific location + category
python main.py discover --location "New York" --category tech --count 15

# Batch mode (multiple cities x categories)
python main.py discover --batch

# JSON output (for agent integration)
python main.py discover --location "Miami" --category crypto --output json

Scrape Profiles

# Scrape from a queue file
python main.py scrape data/queue/New_York_tech_20260217_120000.json

# Scrape a single profile
python main.py scrape --username elonmusk

# Scrape multiple usernames (manual list)
python main.py scrape --usernames user1,user2,user3 --category tech

# Headless mode
python main.py scrape --username nasa --headless

# JSON output
python main.py scrape --username elonmusk --output json

List Queue Files

python main.py list

Export Data

# Export to both JSON and CSV
python main.py export --format both

# JSON only
python main.py export --format json

# CSV only (produces profiles CSV + tweets CSV)
python main.py export --format csv

Directory Structure

twitter-scraper/
├── main.py                  # CLI entry point
├── scraper.py               # Core TwitterScraper class
├── discovery.py             # Profile discovery (Google/DuckDuckGo)
├── anti_detection.py        # Anti-detection system
├── requirements.txt
├── SKILL.md
├── config/
│   └── scraper_config.json  # Configuration
├── data/
│   ├── browser_fingerprints.json
│   ├── output/              # Per-profile JSON output
│   └── queue/               # Discovery queue files
└── thumbnails/              # Downloaded media

Data State

Output Schema

Profile JSON (data/output/{username}.json)

{
  "username": "elonmusk",
  "display_name": "Elon Musk",
  "bio": "...",
  "followers": 180000000,
  "following": 800,
  "tweets_count": 45000,
  "is_verified": true,
  "profile_pic_url": "https://...",
  "profile_pic_local": "thumbnails/elonmusk/profile_abc123.jpg",
  "user_location": "Mars & Earth",
  "join_date": "June 2009",
  "website": "https://x.ai",
  "influencer_tier": "mega",
  "category": "tech",
  "scrape_location": "New York",
  "scraped_at": "2026-02-17T12:00:00",
  "recent_tweets": [
    {
      "id": "1234567890",
      "text": "Tweet content...",
      "timestamp": "2026-02-17T10:30:00.000Z",
      "likes": 50000,
      "retweets": 12000,
      "replies": 3000,
      "views": "5.2M",
      "media_urls": ["https://..."],
      "media_local": ["thumbnails/elonmusk/tweet_media_0_def456.jpg"],
      "is_retweet": false,
      "is_reply": false,
      "url": "https://x.com/elonmusk/status/1234567890"
    }
  ]
}

Queue File (data/queue/{location}_{category}_{timestamp}.json)

{
  "location": "New York",
  "category": "tech",
  "total": 15,
  "usernames": ["user1", "user2", "..."],
  "completed": ["user1"],
  "failed": {"user3": "not_found"},
  "current_index": 2,
  "created_at": "2026-02-17T12:00:00",
  "source": "google_api"
}

Influencer Tiers

Configuration (config/scraper_config.json)

Anti-Detection

The scraper uses multiple anti-detection techniques:

• Browser fingerprinting — 4 rotating fingerprint profiles (viewport, user agent, timezone, WebGL, etc.)
• Stealth JavaScript — Hides navigator.webdriver, spoofs plugins/languages/hardware, canvas noise, fake chrome object
• Human behavior simulation — Random delays, mouse movements, scrolling patterns
• Network randomization — Variable timing between requests
• Login wall handling — Automatically dismisses Twitter's login prompts and overlays

Filters

Profiles are automatically skipped if:

• Account doesn't exist or is suspended
• Account is protected (private)
• Followers count < min_followers config value

Notes

• No login required — Only scrapes publicly visible content
• Checkpoint/resume — Queue files track progress; interrupted scrapes can be resumed with --resume
• Rate limiting — Waits 60s on rate limit, stops on daily limit detection
• Twitter selectors — Uses data-testid attributes (stable across UI changes) with fallbacks to aria-label and structural selectors