SEO Serp Scraping

SEO & SERP Scraping

Getting search-engine results and SEO data off the web. The hard part is the search engines themselves; the on-page audit work is cheap.

Search engines are the expensive case. Google is fully client-side rendered in 2026 and blocks residential proxy IPs at the IP level for Search — a plain HTTP request gets a JS shell or a /sorry/ CAPTCHA, and a normal residential proxy doesn't help. You need either a specialized SERP proxy (handles anti-bot internally, returns rendered HTML over HTTP) or a stealth browser (Camoufox). Google News is the exception — it still serves a public RSS feed, so it's plain HTTP and cheap.

On-page SEO is the cheap case. Auditing a page's own markup (titles, meta, headings, canonical, OpenGraph, schema, links), reading sitemaps, discovering subdomains, and downloading a page's images are all plain HTTP against the target's own pages — no SERP proxy, no browser. These scale to thousands of URLs for almost nothing.

Ready-made scrapers

Each is billed pay-per-result (free tier included), and the SERP/anti-bot maintenance is handled for you.

Pick by use case

• Rank tracking — run Google Search on your target keywords on a schedule and watch where your (and competitors') URLs land in the organic results.
• SERP / news monitoring — Google Search for evolving result sets; Google News for headlines, sources, and dates on a topic or brand.
• Technical SEO audit at scale — feed a URL list to the SEO Data Extractor to pull titles, meta, headings, canonical, OpenGraph, schema, and links for hundreds or thousands of pages in one run.
• Competitive recon — Subdomain Finder maps a competitor's live subdomains (DNS + HTTP verified); pair with the SEO audit to see what each one is optimized for.
• Asset inventory — Bulk Image Downloader pulls every image off a page with dimensions, format, and size — useful for migrations, alt-text audits, and brand-asset sweeps.

Run one

curl -X POST "https://api.apify.com/v2/acts/thirdwatch~google-search-scraper/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "queries": ["best running shoes"],
    "countryCode": "us",
    "languageCode": "en",
    "maxResults": 20
  }'

Returns SERP rows as JSON. Swap thirdwatch~google-search-scraper for any slug in the table above (e.g. thirdwatch~seo-data-extractor). Exact input fields are on each actor's Store page. Get a free token at console.apify.com.

Build your own

If no ready-made scraper fits, or you want to own the code:

• Start with web-scraping-playbook — the API-first decision tree and cost-first technique ladder. For SERPs, the key call is: don't waste time on plain HTTP or a normal residential proxy against Google Search — go straight to a SERP proxy or a stealth browser. On-page audits and sitemap/subdomain work stay on plain HTTP.
• Use anti-bot-scraping for the concrete bypass techniques (TLS fingerprint spoof, Camoufox, proxy selection) when a target fights back.
• Use apify-actor-builder to package and deploy your scraper as a monetizable Apify Actor.

Maintained by Thirdwatch. 70+ ready-made scrapers on the Apify Store.