Dfs Onpage API

DataForSEO OnPage API

Comprehensive on-page SEO analysis with support for instant single-page checks and full website crawls.

Overview

This skill provides access to the DataForSEO OnPage API for technical SEO analysis. It supports two primary workflows:

1. Instant Analysis - Quick single-page analysis without task creation (ideal for spot checks)
2. Full Site Crawls - Comprehensive website audits with task-based crawling (ideal for complete technical audits)

Quick Decision Guide

Use instant_pages.py when:

• Analyzing a single page quickly
• Need immediate results without waiting for crawl
• Checking page performance, Core Web Vitals, or specific technical issues
• No task management needed

Use task_post.py + retrieval scripts when:

• Auditing an entire website
• Need bulk data for multiple pages
• Want to identify site-wide issues (duplicates, broken links, redirect chains)
• Require comprehensive technical SEO audit

Instant Single-Page Analysis

instant_pages.py

Performs real-time analysis of individual web pages without task creation. Returns immediate results.

Basic usage:

python scripts/instant_pages.py --url https://example.com/page

With Core Web Vitals (browser rendering):

python scripts/instant_pages.py \
  --url https://example.com/page \
  --enable-browser-rendering \
  --load-resources

With custom output path:

python scripts/instant_pages.py \
  --url https://example.com/page \
  --enable-browser-rendering \
  --output output/onpage_instant_example.json

Key Parameters:

• --url (required): Target page URL
• --enable-browser-rendering: Measure Core Web Vitals (CLS, LCP, FID)
• --enable-javascript: Execute page JavaScript
• --load-resources: Load images, scripts, stylesheets
• --store-raw-html: Save HTML for later analysis
• --check-spell: Enable spelling verification
• --browser-preset: Device type (desktop, mobile, tablet)
• --output: Custom output path

Returns:

• Page metrics (status code, load time, size)
• OnPage score (0-100)
• Core Web Vitals (if browser rendering enabled)
• SEO elements (title, meta, headings)
• Technical issues (missing tags, broken links, etc.)

Full Website Crawl Workflow

Step 1: Create Crawl Task (task_post.py)

Initiates a website crawl. Returns a task ID for retrieving results.

Basic crawl:

python scripts/task_post.py \
  --target example.com \
  --max-crawl-pages 500

Advanced crawl with Core Web Vitals:

python scripts/task_post.py \
  --target example.com \
  --max-crawl-pages 1000 \
  --enable-browser-rendering \
  --load-resources \
  --calculate-keyword-density \
  --check-spell

Sitemap-based crawl:

python scripts/task_post.py \
  --target example.com \
  --max-crawl-pages 500 \
  --respect-sitemap \
  --crawl-sitemap-only

With priority URLs:

python scripts/task_post.py \
  --target example.com \
  --max-crawl-pages 500 \
  --priority-urls "https://example.com/important,https://example.com/page2"

Key Parameters:

• --target (required): Domain name (without https:// or www.)
• --max-crawl-pages (required): Maximum pages to crawl
• --start-url: Initial URL to start crawling
• --max-crawl-depth: Maximum link depth
• --crawl-delay: Delay between requests (ms)
• --enable-browser-rendering: Enable Core Web Vitals
• --enable-javascript: Execute page JavaScript
• --load-resources: Load images, scripts, stylesheets
• --store-raw-html: Store HTML for later retrieval
• --respect-sitemap: Follow sitemap page order
• --custom-sitemap: Alternative sitemap URL
• --crawl-sitemap-only: Limit to sitemap pages only
• --calculate-keyword-density: Compute keyword frequency
• --check-spell: Enable spell-checking
• --priority-urls: Comma-separated priority URLs (max 20)
• --output: Custom output path

Returns:

• Task ID (save this for retrieving results)
• Cost information

Example output:

Task created successfully!
Task ID: abc123-def456-ghi789

Step 2: Check Task Status (tasks_ready.py)

Check which tasks are completed and ready for data retrieval.

python scripts/tasks_ready.py

Output:

• List of completed tasks with IDs
• Pages crawled for each task
• Retrieval command examples

Step 3: Retrieve Results

Once the task is completed, use the task ID to retrieve different types of data.

Get Summary (summary.py)

Website-level summary with overall metrics and issue counts.

python scripts/summary.py --id abc123-def456-ghi789

With custom output path:

python scripts/summary.py \
  --id abc123-def456-ghi789 \
  --output output/onpage_summary.json

Returns:

• Crawl status (pages crawled, pages in queue)
• Domain info (CMS, server, SSL status)
• Page metrics (OnPage score, links, duplicates, broken links)
• Issue summary (counts by type)

Get Pages (pages.py)

Detailed page-level data with filtering and pagination support.

Basic usage:

python scripts/pages.py --id abc123-def456-ghi789

Filter pages with issues:

python scripts/pages.py \
  --id abc123-def456-ghi789 \
  --filter "checks.no_h1_tag,=,true"

Filter by status code:

python scripts/pages.py \
  --id abc123-def456-ghi789 \
  --filter "status_code,=,404"

Filter by OnPage score:

python scripts/pages.py \
  --id abc123-def456-ghi789 \
  --filter "onpage_score,<,50" \
  --order-by "onpage_score,asc"

Multiple filters (AND logic):

python scripts/pages.py \
  --id abc123-def456-ghi789 \
  --filter "status_code,=,200" \
  --filter "onpage_score,<,70" \
  --limit 1000

Pagination:

python scripts/pages.py \
  --id abc123-def456-ghi789 \
  --limit 1000 \
  --offset 1000

Key Parameters:

• --id (required): Task ID
• --limit: Max pages to return (default 100, max 1000)
• --offset: Skip initial results for pagination
• --filter: Filter condition (field,operator,value) - can specify multiple
• --order-by: Sort criteria (field,direction) - can specify multiple
• --output: Custom output path

Common filters:

• status_code,=,404 - Find 404 pages
• onpage_score,<,50 - Low-quality pages
• checks.no_h1_tag,=,true - Missing H1
• checks.duplicate_title,=,true - Duplicate titles
• checks.broken_links,=,true - Pages with broken links

Get Links (links.py)

Internal and external link data.

python scripts/links.py --id abc123-def456-ghi789

Filter broken links:

python scripts/links.py \
  --id abc123-def456-ghi789 \
  --filter "broken,=,true"

Get Resources (resources.py)

Website resources (images, scripts, stylesheets).

python scripts/resources.py --id abc123-def456-ghi789

Filter by resource type:

python scripts/resources.py \
  --id abc123-def456-ghi789 \
  --filter "resource_type,=,image"

Get Duplicate Tags (duplicate_tags.py)

Pages with duplicate title or meta description tags.

python scripts/duplicate_tags.py --id abc123-def456-ghi789

Returns:

• Groups of pages sharing the same title or meta description
• Useful for identifying duplicate meta tag issues

Get Duplicate Content (duplicate_content.py)

Pages with similar content.

python scripts/duplicate_content.py --id abc123-def456-ghi789

Returns:

• Groups of pages with similar content
• Useful for identifying thin or duplicate content issues

Get Non-Indexable Pages (non_indexable.py)

Pages blocked from search engine indexing.

python scripts/non_indexable.py --id abc123-def456-ghi789

Returns:

• Pages blocked by robots.txt, meta noindex, or other mechanisms
• Reason for each non-indexable page

Get Redirect Chains (redirect_chains.py)

Multiple redirect issues.

python scripts/redirect_chains.py --id abc123-def456-ghi789

Returns:

• Redirect chains with start and end URLs
• Chain length for each redirect sequence
• Useful for identifying redirect inefficiencies

Common Workflows

1. Quick Page Audit

Analyze a single page for technical issues:

python scripts/instant_pages.py \
  --url https://example.com/page \
  --enable-browser-rendering \
  --load-resources \
  --check-spell

2. Full Site Technical Audit

Comprehensive website audit:

# Step 1: Create crawl task
python scripts/task_post.py \
  --target example.com \
  --max-crawl-pages 1000 \
  --enable-browser-rendering \
  --calculate-keyword-density

# Step 2: Wait for completion, then check status
python scripts/tasks_ready.py

# Step 3: Get summary
python scripts/summary.py --id <task-id>

# Step 4: Get detailed pages
python scripts/pages.py --id <task-id> --limit 1000

# Step 5: Get specific issues
python scripts/duplicate_tags.py --id <task-id>
python scripts/links.py --id <task-id> --filter "broken,=,true"
python scripts/redirect_chains.py --id <task-id>

3. Find Low-Quality Pages

Identify pages with low OnPage scores:

python scripts/pages.py \
  --id <task-id> \
  --filter "onpage_score,<,60" \
  --order-by "onpage_score,asc" \
  --limit 1000

4. Audit Site-Wide Issues

Get all pages with specific technical issues:

# Missing H1 tags
python scripts/pages.py \
  --id <task-id> \
  --filter "checks.no_h1_tag,=,true"

# Duplicate titles
python scripts/pages.py \
  --id <task-id> \
  --filter "checks.duplicate_title,=,true"

# Large page size
python scripts/pages.py \
  --id <task-id> \
  --filter "size,>,1000000" \
  --order-by "size,desc"

Output Structure

All scripts save data to:

• output/ under the current working directory by default
• A custom path when using --output

File naming:

• onpage_instant_<domain>.json - Instant analysis
• onpage_task_<domain>.json - Task creation response
• onpage_summary_<task-id>.json - Summary data
• onpage_pages_<task-id>.json - Pages data
• onpage_links_<task-id>.json - Links data
• onpage_resources_<task-id>.json - Resources data
• onpage_duplicate_tags_<task-id>.json - Duplicate tags
• onpage_duplicate_content_<task-id>.json - Duplicate content
• onpage_non_indexable_<task-id>.json - Non-indexable pages
• onpage_redirect_chains_<task-id>.json - Redirect chains

API Limits

Pricing Notes

• Charged per crawl task (varies by page count and features)
• Browser rendering adds cost per page
• JavaScript execution adds cost per page
• Instant pages charged per request
• Cost displayed after each operation

Environment Variables

Requires in .env:

DATAFORSEO_LOGIN=your_login
DATAFORSEO_PASSWORD=your_password

The scripts automatically load credentials from:

1. Repo root .env (4 directories up from script)
2. Current working directory .env

Troubleshooting

"Task not found" error:

• Task may still be processing - wait and retry
• Use tasks_ready.py to check task status
• Verify task ID is correct

"HTTP Error 401":

• Check DATAFORSEO_LOGIN and DATAFORSEO_PASSWORD in .env
• Verify credentials are correct

"No results returned":

• Task may have failed during crawl
• Check summary for crawl status
• Verify target domain is accessible

"Rate limit exceeded":

• API enforces 2000 requests/minute
• Scripts handle rate limiting automatically
• Contact DataForSEO if persistent

Key Metrics Explained

OnPage Score (0-100):

• Composite score reflecting overall page optimization
• Higher = better optimized
• < 60 typically indicates issues requiring attention

Core Web Vitals (requires browser rendering):

• CLS (Cumulative Layout Shift): Visual stability (lower is better, < 0.1 good)
• LCP (Largest Contentful Paint): Load performance (lower is better, < 2500ms good)
• FID (First Input Delay): Interactivity (lower is better, < 100ms good)

Common Checks:

• no_h1_tag: Missing H1 heading
• duplicate_title: Title tag appears on multiple pages
• duplicate_description: Meta description appears on multiple pages
• broken_links: Page contains broken links
• large_page_size: Page size exceeds threshold
• slow_page_load: Page load time exceeds threshold
• no_image_alt: Images missing alt attributes
• deprecated_html: Uses deprecated HTML tags