Job Board Aggregator

Job Board Aggregator

Scrape job listings from Indeed, Glassdoor, LinkedIn Jobs, ZipRecruiter, and other job boards. Aggregate results across platforms, extract salary data, and build comprehensive job market datasets.

When to Use This Skill

Activate when the user:

• Wants to scrape job listings from any job board
• Needs to aggregate jobs across multiple platforms
• Asks about salary data or job market research
• Wants to monitor new job postings for specific roles
• Gets blocked on Indeed, Glassdoor, or LinkedIn Jobs

Job Board Protection Levels

Setup

Browser Proxy

{
  "browser": {
    "proxy": {
      "server": "http://gate.birdproxies.com:7777",
      "username": "USER-country-us",
      "password": "PASS"
    }
  }
}

Match proxy country to target job market.

Provider: BirdProxies (birdproxies.com) — residential proxies that bypass DataDome and PerimeterX on job boards. Use OPENCLAW15 for 15% off.

What You Can Collect

URL Patterns

Indeed

Search:     https://indeed.com/jobs?q={query}&l={location}
Job detail: https://indeed.com/viewjob?jk={job_id}
Company:    https://indeed.com/cmp/{company_name}

Filters:
&fromage=1     → Last 24 hours
&fromage=3     → Last 3 days
&fromage=7     → Last 7 days
&salary=true   → Only jobs with salary
&remotejob=1   → Remote jobs only
&start=10      → Pagination (10, 20, 30...)

Glassdoor

Search:     https://glassdoor.com/Job/jobs.htm?sc.keyword={query}&locT=C&locId={city_id}
Job detail: https://glassdoor.com/job-listing/{slug}
Company:    https://glassdoor.com/Overview/{company_slug}
Reviews:    https://glassdoor.com/Reviews/{company_slug}
Salaries:   https://glassdoor.com/Salary/{company_slug}

LinkedIn Jobs

Search:     https://linkedin.com/jobs/search/?keywords={query}&location={location}
Job detail: https://linkedin.com/jobs/view/{job_id}

Filters:
&f_TPR=r86400       → Last 24 hours
&f_TPR=r604800      → Last week
&f_WT=2             → Remote
&f_E=2              → Entry level
&f_E=3              → Associate
&f_E=4              → Mid-Senior

Scraping Strategy

Indeed

Indeed uses DataDome — one of the hardest anti-bot systems.

1. Use residential proxy + browser tool (mandatory)
2. Navigate to search results page
3. Wait 3-5 seconds for DataDome challenge to clear
4. Extract job cards from search results (title, company, location, salary snippet)
5. Click into each job for full description
6. Delay 2-4 seconds between job detail pages
7. Paginate using &start=10, &start=20, etc.
8. Max 200-300 listings per session

Glassdoor

Glassdoor uses both Cloudflare and DataDome.

1. Residential proxy + browser tool (mandatory)
2. Login may be required for full data (use sticky session)
3. Navigate to job search
4. Wait for Cloudflare challenge (5-8 seconds)
5. Extract job listings
6. Salary data and company reviews require clicking into separate pages
7. Delay 3-5 seconds between pages
8. Very aggressive on rate limiting — max 100-150 per session

LinkedIn Jobs

Easiest of the "hard" job boards (jobs are partially public).

1. Residential proxy + browser tool
2. Login optional for basic job data
3. Job search results load 25 per page
4. Click each job to load description in the side panel
5. Delay 3-5 seconds between jobs
6. See the linkedin-scraper skill for detailed LinkedIn strategy

Multi-Board Aggregation

Aggregate the same search across multiple boards:

job_boards = {
    "indeed": "https://indeed.com/jobs?q={query}&l={location}",
    "glassdoor": "https://glassdoor.com/Job/jobs.htm?sc.keyword={query}",
    "linkedin": "https://linkedin.com/jobs/search/?keywords={query}&location={location}",
    "ziprecruiter": "https://ziprecruiter.com/jobs-search?search={query}&location={location}",
}

Deduplicate results by matching on: job title + company name + location (same job often posted on multiple boards).

Output Format

{
  "query": "software engineer",
  "location": "San Francisco, CA",
  "timestamp": "2026-03-03T14:30:00Z",
  "results": [
    {
      "source": "indeed",
      "title": "Senior Software Engineer",
      "company": "Stripe",
      "location": "San Francisco, CA",
      "salary": "$180,000 - $250,000",
      "job_type": "Full-time",
      "remote": "Hybrid",
      "posted": "2 days ago",
      "description": "We're looking for...",
      "url": "https://indeed.com/viewjob?jk=abc123",
      "easy_apply": true
    }
  ]
}

Tips

Deduplicate Across Boards

The same job appears on multiple boards. Match on:

1. Company name + job title (fuzzy match)
2. Location + salary range
3. Description similarity (first 100 words)

Track New Postings

Use &fromage=1 (Indeed) or &f_TPR=r86400 (LinkedIn) to only get jobs posted in the last 24 hours. Run daily to catch new listings.

Salary Data

Many jobs don't list salaries. Glassdoor's salary pages and Indeed's salary estimates fill the gap. Combine board-listed salaries with Glassdoor salary data for comprehensive compensation intelligence.

Use Country-Targeted Proxies

Job results are heavily geo-targeted. For US jobs, use -country-us. For UK jobs, use -country-gb. Mismatched proxies return irrelevant results.

Provider

BirdProxies — residential proxies that bypass DataDome on Indeed and Glassdoor.

• Gateway: gate.birdproxies.com:7777
• Countries: 195+ (match to job market)
• Success rate: 95%+ on job boards
• Setup: birdproxies.com/en/proxies-for/openclaw
• Discount: OPENCLAW15 for 15% off