Lead Scraping & Verification
Goal
Scrape leads using Apify (code_crafter/leads-finder), verify their relevance (industry match > 80%), and save them to a Google Sheet. For large scrapes (1000+ leads), use parallel scraping for 3-5x faster performance.
Inputs
- Industry: The target industry (e.g., "Plumbers", "Software Agencies")
- Location: The target location (e.g., "Texas", "United States", "California"). Scripts auto-format to Apify's required format (US states get ", us" suffix automatically).
- Total Count: The total number of leads desired
Scripts
All scripts are in ./scripts/:
scrape_apify.py - Single scrape, for <1000 leads
scrape_apify_parallel.py - Parallel scraping, for 1000+ leads
classify_leads_llm.py - LLM-based lead classification
enrich_emails.py - Email enrichment via AnyMailFinder
update_sheet.py - Batch sheet updates
read_sheet.py - Read data from Google Sheets
Process
Small Scrapes (<1000 leads)
Test Scrape
python3 ./scripts/scrape_apify.py --query "INDUSTRY" --location "LOCATION" --max_items 25 --no-email-filter --output .tmp/test_leads.json
Verification
- Read
.tmp/test_leads.json
- Check if at least 20/25 (80%) leads match the Industry
- Pass: Proceed to step 3
- Fail: Stop and ask user to refine keywords
Full Scrape
python3 ./scripts/scrape_apify.py --query "INDUSTRY" --location "LOCATION" --max_items TOTAL_COUNT --no-email-filter --output .tmp/leads.json
[Optional] LLM Classification (for complex niches)
python3 ./scripts/classify_leads_llm.py .tmp/leads.json --classification_type product_saas --output .tmp/classified_leads.json
Upload to Google Sheet
python3 ./scripts/update_sheet.py .tmp/leads.json --title "Leads - INDUSTRY"
Enrich Missing Emails
python3 ./scripts/enrich_emails.py SHEET_URL
Large Scrapes (1000+ leads)
Test Scrape (same as above with 25 items)
Parallel Full Scrape
python3 ./scripts/scrape_apify_parallel.py \
--query "INDUSTRY" \
--total_count 4000 \
--location "United States" \
--strategy regions \
--no-email-filter
Geographic partitioning is automatic:
- United States: 4-way (Northeast, Southeast, Midwest, West)
- EU/Europe: 4-way (Western, Southern, Northern, Eastern)
- UK: 4-way (SE England, N England, Scotland/Wales, SW England)
- Canada: 4-way (Ontario, Quebec, West, Atlantic)
- Australia: 4-way (NSW, VIC/TAS, QLD, WA/SA)
Continue with steps 4-6 from small scrapes
Outputs
The ONLY deliverable is the Google Sheet URL. Local JSON files in .tmp/ are temporary intermediates.
Edge Cases
- No leads found: Ask user to broaden search
- API Error: Check credentials in
.env
- Low quality classifications: If >80% "unclear", improve scrape keywords
Environment
Requires in .env:
APIFY_API_TOKEN=your_token
GOOGLE_APPLICATION_CREDENTIALS=path/to/credentials.json
ANTHROPIC_API_KEY=your_key
ANYMAILFINDER_API_KEY=your_key
Schema
Inputs
| Name |
Type |
Required |
Description |
industry |
string |
Yes |
Target industry (e.g., 'Plumbers', 'Software Agencies') |
location |
string |
Yes |
Target location (e.g., 'Texas', 'United States') |
total_count |
integer |
Yes |
Total number of leads desired |
classification_type |
string |
No |
LLM classification type (e.g., 'product_saas') |
Outputs
| Name |
Type |
Description |
sheet_url |
string |
Google Sheet URL with scraped leads |
lead_count |
integer |
Number of leads found |
Credentials
| Name |
Source |
APIFY_API_TOKEN |
.env |
GOOGLE_APPLICATION_CREDENTIALS |
.env |
ANTHROPIC_API_KEY |
.env |
ANYMAILFINDER_API_KEY |
.env |
Composable With
Skills that chain well with this one: classify-leads, casualize-names, instantly-campaigns, onboarding-kickoff
Cost
$0.01-0.02 per lead + $0.30/1K for classification
1---2name: scrape-leads3description: Scrape and verify business leads using Apify, classify with LLM, enrich emails, and save to Google Sheets. Use when user asks to find leads, scrape businesses, generate prospect lists, or build lead databases for any industry or location.4argument-hint: "[industry] [location] [count]"5allowed-tools: Bash, Read, Write, Edit, Glob, Grep6metadata:7 source: https://skillmd.com/skills/aiagentwithdhruv/scrape-leads8---910# Lead Scraping & Verification1112## Goal13Scrape leads using Apify (`code_crafter/leads-finder`), verify their relevance (industry match > 80%), and save them to a Google Sheet. For large scrapes (1000+ leads), use parallel scraping for 3-5x faster performance.1415## Inputs16- **Industry**: The target industry (e.g., "Plumbers", "Software Agencies")17- **Location**: The target location (e.g., "Texas", "United States", "California"). Scripts auto-format to Apify's required format (US states get ", us" suffix automatically).18- **Total Count**: The total number of leads desired1920## Scripts21All scripts are in `./scripts/`:22- `scrape_apify.py` - Single scrape, for <1000 leads23- `scrape_apify_parallel.py` - Parallel scraping, for 1000+ leads24- `classify_leads_llm.py` - LLM-based lead classification25- `enrich_emails.py` - Email enrichment via AnyMailFinder26- `update_sheet.py` - Batch sheet updates27- `read_sheet.py` - Read data from Google Sheets2829## Process3031### Small Scrapes (<1000 leads)32331. **Test Scrape**34 ```bash35 python3 ./scripts/scrape_apify.py --query "INDUSTRY" --location "LOCATION" --max_items 25 --no-email-filter --output .tmp/test_leads.json36 ```37382. **Verification**39 - Read `.tmp/test_leads.json`40 - Check if at least 20/25 (80%) leads match the Industry41 - **Pass**: Proceed to step 342 - **Fail**: Stop and ask user to refine keywords43443. **Full Scrape**45 ```bash46 python3 ./scripts/scrape_apify.py --query "INDUSTRY" --location "LOCATION" --max_items TOTAL_COUNT --no-email-filter --output .tmp/leads.json47 ```48494. **[Optional] LLM Classification** (for complex niches)50 ```bash51 python3 ./scripts/classify_leads_llm.py .tmp/leads.json --classification_type product_saas --output .tmp/classified_leads.json52 ```53545. **Upload to Google Sheet**55 ```bash56 python3 ./scripts/update_sheet.py .tmp/leads.json --title "Leads - INDUSTRY"57 ```58596. **Enrich Missing Emails**60 ```bash61 python3 ./scripts/enrich_emails.py SHEET_URL62 ```6364### Large Scrapes (1000+ leads)65661. **Test Scrape** (same as above with 25 items)67682. **Parallel Full Scrape**69 ```bash70 python3 ./scripts/scrape_apify_parallel.py \71 --query "INDUSTRY" \72 --total_count 4000 \73 --location "United States" \74 --strategy regions \75 --no-email-filter76 ```7778 Geographic partitioning is automatic:79 - **United States**: 4-way (Northeast, Southeast, Midwest, West)80 - **EU/Europe**: 4-way (Western, Southern, Northern, Eastern)81 - **UK**: 4-way (SE England, N England, Scotland/Wales, SW England)82 - **Canada**: 4-way (Ontario, Quebec, West, Atlantic)83 - **Australia**: 4-way (NSW, VIC/TAS, QLD, WA/SA)84853. Continue with steps 4-6 from small scrapes8687## Outputs88**The ONLY deliverable is the Google Sheet URL.** Local JSON files in `.tmp/` are temporary intermediates.8990## Edge Cases91- **No leads found**: Ask user to broaden search92- **API Error**: Check credentials in `.env`93- **Low quality classifications**: If >80% "unclear", improve scrape keywords9495## Environment96Requires in `.env`:97```98APIFY_API_TOKEN=your_token99GOOGLE_APPLICATION_CREDENTIALS=path/to/credentials.json100ANTHROPIC_API_KEY=your_key101ANYMAILFINDER_API_KEY=your_key102```103104---105106## Schema107108### Inputs109| Name | Type | Required | Description |110|------|------|----------|-------------|111| `industry` | string | Yes | Target industry (e.g., 'Plumbers', 'Software Agencies') |112| `location` | string | Yes | Target location (e.g., 'Texas', 'United States') |113| `total_count` | integer | Yes | Total number of leads desired |114| `classification_type` | string | No | LLM classification type (e.g., 'product_saas') |115116### Outputs117| Name | Type | Description |118|------|------|-------------|119| `sheet_url` | string | Google Sheet URL with scraped leads |120| `lead_count` | integer | Number of leads found |121122### Credentials123| Name | Source |124|------|--------|125| `APIFY_API_TOKEN` | .env |126| `GOOGLE_APPLICATION_CREDENTIALS` | .env |127| `ANTHROPIC_API_KEY` | .env |128| `ANYMAILFINDER_API_KEY` | .env |129130### Composable With131Skills that chain well with this one: `classify-leads`, `casualize-names`, `instantly-campaigns`, `onboarding-kickoff`132133### Cost134$0.01-0.02 per lead + $0.30/1K for classification