Brightdata Indeed

BrightData Indeed Scraper

Indeed data collection using BrightData's Datasets API. All operations save results to disk and return summaries to context.

Quick Start

Single Job Listing:

python skills/brightdata-indeed/scripts/fetch.py \
  --endpoint jobs \
  --input "https://www.indeed.com/viewjob?jk=a4d97fa27223cc84"

Job Search by Keyword:

python skills/brightdata-indeed/scripts/fetch.py \
  --endpoint jobs-search \
  --country US \
  --domain indeed.com \
  --keyword "software engineer" \
  --location "New York, NY"

Company Profile:

python skills/brightdata-indeed/scripts/fetch.py \
  --endpoint companies \
  --input "https://www.indeed.com/cmp/Google"

Available Endpoints

Job Listings (Dataset: gd_l4dx9j9sscpvs7no2)

Company Info (Dataset: gd_l7qekxkv2i7ve6hx1s)

Parameters

Common Parameters

Job Search Parameters (jobs-search)

Company Industry Parameters (companies-industry)

Company Search Parameters (companies-search)

Usage Examples

Job Listings

Single job by URL:

python skills/brightdata-indeed/scripts/fetch.py \
  --endpoint jobs \
  --input "https://www.indeed.com/viewjob?jk=a4d97fa27223cc84"

Multiple jobs (comma-separated):

python skills/brightdata-indeed/scripts/fetch.py \
  --endpoint jobs \
  --input "https://www.indeed.com/viewjob?jk=abc123,https://www.indeed.com/viewjob?jk=def456"

Batch from file:

python skills/brightdata-indeed/scripts/fetch.py \
  --endpoint jobs \
  --input job_urls.txt \
  --batch

Job Search

Basic search:

python skills/brightdata-indeed/scripts/fetch.py \
  --endpoint jobs-search \
  --country US \
  --domain indeed.com \
  --keyword "data scientist" \
  --location "San Francisco"

German job search:

python skills/brightdata-indeed/scripts/fetch.py \
  --endpoint jobs-search \
  --country DE \
  --domain de.indeed.com \
  --keyword "softwareentwickler" \
  --location "Berlin"

French job search:

python skills/brightdata-indeed/scripts/fetch.py \
  --endpoint jobs-search \
  --country FR \
  --domain fr.indeed.com \
  --keyword "product manager" \
  --location "Paris"

Jobs from Company

Company jobs page:

python skills/brightdata-indeed/scripts/fetch.py \
  --endpoint jobs-by-company \
  --input "https://www.indeed.com/cmp/Semrush/jobs"

Company Information

Single company:

python skills/brightdata-indeed/scripts/fetch.py \
  --endpoint companies \
  --input "https://www.indeed.com/cmp/Google"

Search companies by keyword:

python skills/brightdata-indeed/scripts/fetch.py \
  --endpoint companies-search \
  --keyword "Tesla"

Companies by industry:

python skills/brightdata-indeed/scripts/fetch.py \
  --endpoint companies-industry \
  --industry "Drug & Health Stores" \
  --state "California"

Batch Operations

JSON Format

{
  "urls": [
    "https://www.indeed.com/viewjob?jk=abc123",
    "https://www.indeed.com/viewjob?jk=def456"
  ]
}

Job Search Batch

{
  "searches": [
    {"country": "US", "domain": "indeed.com", "keyword_search": "software engineer", "location": "New York"},
    {"country": "DE", "domain": "de.indeed.com", "keyword_search": "developer", "location": "Berlin"},
    {"country": "FR", "domain": "fr.indeed.com", "keyword_search": "data scientist", "location": "Paris"}
  ]
}

Async Mode

For large batches, use async mode:

# Submit async request
python skills/brightdata-indeed/scripts/fetch.py \
  --endpoint jobs \
  --input job_urls.json \
  --batch \
  --mode async

# Check status and retrieve results
python skills/brightdata-indeed/scripts/status.py \
  --snapshot-id <snapshot_id> \
  --save

Output Format

{
  "success": true,
  "endpoint": "jobs-search",
  "mode": "sync",
  "record_count": 25,
  "output_file": "output/brightdata_indeed_jobs_search_20250125T123456.json",
  "summary": "Successfully scraped 25/25 Indeed jobs-search",
  "stats": {
    "total_inputs": 1,
    "successful": 25,
    "failed": 0
  }
}

Job Listing Output Fields

Key fields returned for job listings:

• job_title - Title of the job position
• company_name - Name of the hiring company
• location - Job location
• salary_formatted - Formatted salary range
• job_type - Employment type (full-time, part-time, etc.)
• description_text - Full job description
• benefits - Job benefits offered (array)
• qualifications - Required qualifications (array)
• date_posted_parsed - Standardized posting date

Company Output Fields

Key fields returned for companies:

• name - Company name
• description - Company overview
• website - Official website
• industry - Industry classification
• company_size - Employee count range
• revenue - Revenue information
• headquarters - HQ location
• reviews_count - Number of reviews
• salaries - Salary information (object)
• benefits - Benefits offered (array)
• work_happiness - Employee satisfaction metrics (array)

API Configuration

Requires BRIGHTDATA_API_TOKEN in .env:

BRIGHTDATA_API_TOKEN=your_api_token_here

Error Handling

Partial success is supported - if some inputs fail, successful ones are still saved:

{
  "success": true,
  "stats": {
    "total_inputs": 10,
    "successful": 8,
    "failed": 2
  }
}

Limitations

• Up to 1,000 jobs per input for job searches
• Rate limits depend on BrightData plan
• Average response time: ~2s per input
• Pricing: ~$0.0015 per record