Octoparse Ultimate Scraper

Octoparse data collection

The Octoparse platform's collection capability, exposed for an agent to drive through the MCP server.

Collection reaches the user through more than one path, and Step 1 picks between them. Preset templates are the largest today; the tasks the user built themselves are already on the account and cost nothing to look up; managed datasets beat templates where they overlap; agent-generated tasks are planned for the long tail. Route to the capability that fits the request, not to templates by default.

Division of responsibility — read this before anything else:

• This skill routes. Which capability applies, which template fits within it, which pairs are real chains, what to avoid.
• The MCP service executes and holds truth about schemas. Input/output fields, sourceTree, pricing, and current availability come from search_templates at run time, never from memory.
• data/catalog.json is a routing projection only — it is regenerated from upstream, carries no input or output schemas, and is read by this repository's checks rather than by you at run time.

Never assemble parameters from this skill's prose or from a remembered field name. Always read the live inputSchema first.

Prerequisites

Octoparse MCP server connected and authorized. If search_templates fails with an auth error, use the octoparse-mcp-setup skill.

Workflow

Step 1 — Pick the capability

Octoparse exposes distinct capabilities that share no tools. Pick before doing anything else.

Check the account before assuming a request needs a template. When someone describes a collection they have run before, or names something that sounds like their own configuration, search_tasks is one call and settles it.

The dataset capability is expanding beyond those two platforms, and for a platform it covers it is the better choice, not a fallback — typed schema, server-side filtering, and fields no template exposes. Check references/dataset-capability.md before assuming a platform is template-only.

Long-tail sites have no path today. An agent-generated task capability is planned to cover sites the preset library does not reach. Until it ships, an uncovered site is a genuine "no" — say so plainly and stop. Do not improvise by bending a nearby template onto a site it was not built for: it will either fail or return the wrong page's data, and it bills either way.

Before concluding a site is uncovered, exhaust the real options: the workflow guide's curated ids, a semantic search phrased as full intent, and the dataset capability's current platform list. Semantic recall is uneven, so a thin result set is not proof.

Step 2 — Route to a workflow guide

Guides are organised by what the user is trying to accomplish, not by the library's category tags. Read the matching guide before selecting a template — each carries a curated shortlist, the market coverage, and the traps for that domain.

Presenting a multi-template workflow? Read references/chaining.md first. A chain is only real when the upstream URL type matches the downstream input type, and the upstream pass bills whether or not the downstream one accepts its output.

Two boundaries worth knowing, because requests land on them constantly:

• Price monitoring vs market research — detail pages vs listing pages. "Watch these 20 products" is the first; "what sells in this category" is the second.
• Lead generation vs company research — a contactable list vs a judgement about a company. The same directory site serves both; the difference is which fields matter.

No guide fits, or the guide's shortlist misses the target site — search the full library:

search_templates(query="<site + data + intent, in one sentence>", limit=10)

Describe the complete intent, not a keyword. "collect product name price and rating from Amazon Japan search results" beats "amazon".

Filter the results yourself on executionMode. It is an array; only templates containing "Cloud" can run through MCP. Local-only templates are returned by both search modes despite what the API docs claim — roughly a third of a typical result set. Ranking by score alone will hand you a template that cannot run.

Semantic recall is uneven. A narrow query can return a handful of results and miss the obvious best match, so treat a thin result set as a failed search, not as evidence that nothing exists — fall back to the workflow guide's curated ids and look them up directly.

Step 3 — Fetch the live schema

Exact lookup on the chosen template:

search_templates(id=<template_id>)

This is the authoritative source for everything you need to run the template. Take three things from it:

• templateName — execute_task is keyed by this, and it is a slug (contact-details-scraper), not the display name. Copy it verbatim, including when it is a placeholder string like aaaaaaaaa.
• inputSchema[].field — the exact parameters keys. Field names here are normalised and do not match any other source; never carry a field name over from documentation, memory, or a previous template.
• executionMode — confirm it contains "Cloud". Exact lookup returns local-only templates too.
• pricing — the current per-line cost. Workflow guides describe cost only in relative terms because absolute figures decay; quote the actual number from here.

The account tier a template requires is not in the service response. Workflow guides carry it from a snapshot, so treat a STANDARD marking as a warning to raise with the user rather than a fact to assert — and if a run fails on entitlement despite the guide saying FREE, trust the failure.

If exact lookup returns nothing, the template is not currently served by MCP. Say so and stop; do not substitute a different template silently, and do not retry.

If templates/<id>-*/LIMITATIONS.md exists for the chosen template, read it. A few high-traffic templates carry one, covering operational behaviour that no schema exposes — result caps, how the template splits a region, whether one task can span several locations. Most templates have none; skip this step when the file is absent.

Step 4 — Confirm scope, then run

Skip confirmation for small lookups. For anything that produces a list, settle target site/region, result size, and required fields first — templates bill per output line, and MCP usage on a free account draws on an allowance of 2,000 rows per month.

Size the run in rows, not dollars: that is the limit users actually hit, it applies to free and paid templates alike, and no tool can report how much of the allowance is left. See references/gotchas.md.

execute_task(
  templateName="<from Step 3>",
  parameters="<JSON object string keyed by inputSchema[].field>",
  taskName="<unique, descriptive>"
)

parameters is a JSON object encoded as a string, not an object. Keys are exactly inputSchema[].field. Value shape follows each field's uiType:

When a field carries valueFormat: "string[]", that settles it regardless of uiType. Unfamiliar and empty uiType values do occur; fall back to valueFormat, then to type.

Always pass a unique taskName. The call blocks up to 45 seconds; if the client times out first, the run still exists and search_tasks(keyword="<taskName>") is the only way back to its taskId.

Source-backed fields: when inputSchema[].sourceBacked is true, the allowed values come from sourceTree in the Step 3 response. sourceTree resolves every level at once — a root option carries its dependent field's options inside children — so read both levels there and pass the option key values (not the labels). Doing that avoids the round-trip entirely.

If a dependent selection is still missing, execute_task returns input_required plus the next sourceOptions; add the chosen key and call again. This is a normal handshake, not an error.

Returns taskId, lotNo, collectedRows, and status (running / completed / stopped). Keep taskId and lotNo — export needs both.

Step 5 — Poll only if still running

Small jobs finish inside the 45-second window and come back completed — skip straight to export. Poll only when execute_task returned running:

get_task_status(taskId="<taskId>")

collectedRows defaults to 0, so it cannot distinguish "running, nothing yet" from "finished, found nothing". Read status for that.

Poll at widening intervals — 10s, 30s, 60s, then 60s thereafter — for at most 8 checks.

If status is still running at that point, stop polling and hand back to the user: report the taskId, lotNo, and current collectedRows, and say the run continues in the cloud and can be exported later. Do not keep polling; a large collection can take far longer than a session, and the task survives independently of this conversation.

Step 6 — Export

export_data(taskId="<taskId>", lotNo="<lotNo>", page=1, pageSize=20)

Paged; pageSize maxes at 100. There is no file-format parameter — the tool returns rows.

At 50+ total rows, do not page through the tool. The response carries directAccess with a short-lived signed link and a ready curlTemplate that writes to directAccess.outputFile. Run the curl, then read the file. Paging large result sets through the tool floods context for no benefit.

Step 7 — Report

State: row count, where the data landed, the fields that matter for the user's goal, and the honest limitations from the workflow guide. Suggest the next enrichment step only when the guide documents a real one.

Name output columns from the exported rows, not from outputSchema — the schema under-reports. See references/gotchas.md.

Dataset path

Temu and TikTok Shop bypass templates entirely, using a submit-then-query API with three tools: describe_ecommerce_dataset, ecommerce_data_task, and query_collected_reviews. None of the template rules above apply.

Read references/dataset-capability.md for the sequence, dataset names, limits, and traps.

Tasks the user already has

search_tasks returns every task on the account, not only the ones created through MCP — including tasks the user built in the Octoparse client. Nothing in Steps 2–4 applies here: there is no template to choose, no inputSchema to read, and no execute_task.

search_tasks(keyword="<fragment of the name>")

Returns taskId, taskName, and taskStatusLabel per task. Then pick a path:

They want what was already collected — no new run, no cost:

get_task_status(taskId="<taskId>")   → carries lotNo when a lot exists
export_data(taskId=…, lotNo=…, page=1, pageSize=20)

They want fresh data — this bills exactly like a template run, at the same per-row cost against the same allowance, so confirm scope first:

start_or_stop_task(taskId="<taskId>", action="start")

Then poll and export as in Steps 5 and 6. start_or_stop_task rejects a task that is already in the target state.

Three things to know before searching:

• status cannot find a task that has never run. The filter accepts Running, Stopped, Completed, and Failed, but a configured task that has never run comes back labelled Ready — often exactly the one being asked about. Search by keyword and filter the results yourself.
• Names repeat. The client does not enforce unique task names, and several tasks with the same name in different states is normal. Show the matches with their status and let the user choose rather than picking the first.
• A saved task's inputs cannot be changed from here. It runs the configuration it was built with. If the user wants different inputs, they edit the task in Octoparse, or the request is a template job after all.

Also the recovery path after a client timeout: search_tasks(keyword="<taskName>") is the only way back to the taskId of a run that started before the call returned.

Troubleshooting

Read references/gotchas.md before the first execute_task of a session. It covers billing traps, the templates that already collect detail pages internally, unusable slugs, misleading language values, and category noise.