Web Scraping Foundations

Web Scraping Foundations

Use When

• Choose the least invasive authorised acquisition path for web data and validate politeness, structure, and failure handling.

Do Not Use When

• Use dataset-discovery first when published data may exist and scraping-engineering-python only after scaling needs are observed.

Inputs

Workflow

1. Confirm authority and check published APIs/datasets.
2. Inspect response HTML, JSON endpoints, and structured data in order.
3. Choose the cheapest working stack and define typed failures.
4. Test representative records; stop on blocks, consent barriers, or unstable selectors.
5. Recover by narrowing scope, slowing requests, or returning a gap; escalate to engineering only when justified.

Outputs

Evidence Produced

Capability Contract

Planning and review default to read-only. Network requests, browser control, account/session use, form submission, persistent storage, or production crawling require explicit authority. Never bypass access controls.

Degraded Mode

Without network, browser, credentials, or target permission, return a design and offline parser test where possible. Mark live politeness, rendering, and extraction checks unassessed.

Decision Rules

Quality Standards

The route is authorised, least invasive, polite, source-traceable, failure-aware, and proven on representative records.

Anti-Patterns

• Starting with a browser. Fix: run the decision tree.
• Ignoring robots/politeness. Fix: assess before requests.
• Retrying all errors. Fix: classify failures.
• Positional selectors. Fix: use semantic selectors.
• Claiming success from one happy record. Fix: test representative variation.

Worked Example

If a page exposes the needed records through a documented JSON endpoint, choose it over browser automation, test null and pagination behaviour, and stop if the service signals blocking.

References

• Politeness and rate limiting
• Troubleshooting
• Browser automation

The single entry skill for web scraping. Encodes the decision tree, stack choices, and orchestration rules. Detailed sub-disciplines live in references/ and are loaded only when the situation demands them.

Companion skill

• scraping-engineering-python — kept separate because it is the Python-specific scaling layer (caching, concurrency, dynamic content, Scrapy framework selection). Load it when the crawl exceeds ~1,000 URLs, requires resumability, or needs concurrent downloading.

Reference index

When the situation matches the trigger, load the named reference verbatim. Do not load all references by default.

1. The decision tree (memorise — never skip)

Run in this order on every job. Stop at the first step that succeeds.

1. Is the data available from a published API or open dataset? Use it. Skip scraping. (See dataset-discovery-and-analysis.)
2. View Source. Is the data in the response HTML? Yes → requests/httpx + BeautifulSoup. Done.
3. DevTools → Network → XHR. Is there a JSON endpoint? Yes → call it directly. Faster and more stable than HTML parsing.
4. Inline JSON in <script> tags? (__NEXT_DATA__, window.__INITIAL_STATE__.) Extract and parse before considering a browser.
5. Does the page require JS interaction (click, scroll, multi-step form)? Only now → load references/browser-automation-playwright.md.
6. Multi-domain crawl with politeness, dedup, pause/resume needs? Load scraping-engineering-python for Scrapy.

Brody's cheapest diagnostic: "View Source → Cmd-F for a known datum." If the datum is visible there, no JS is needed.

2. Always check structured-data shortcuts FIRST

Many sites publish clean structured data:

• JSON-LD in <script type="application/ld+json"> — Schema.org Product/Article/Organization (tools.scraping.extract_jsonld).
• Open Graph + Twitter Cards — og:title, og:image, og:description (tools.scraping.extract_opengraph).
• RSS / Atom feeds — feedparser (tools.scraping.extractors.feeds.parse_feed).
• /sitemap.xml — server-blessed enumeration without link-following.
• /robots.txt — Sitemap declarations + Disallow rules.

A scraper that ignores these layers wastes effort.

3. Stack choices

4. Pagination patterns

Edge-case probe: try page_size=1000 first — servers often don't validate against UI options.

5. Error taxonomy

Anti-pattern: silent retry on 4xx. Surface, don't bury.

6. Orchestration — what to load and when

new scraping task
├─ load this SKILL.md
├─ ALWAYS load references/politeness-and-ratelimiting.md before issuing requests
├─ Step 2 fails (content differs from browser, or empty)
│   └─ load references/troubleshooting-brody.md
├─ Step 5 fires (JS interaction required)
│   └─ load references/browser-automation-playwright.md
└─ Crawl exceeds 1,000 URLs / needs resumability / multi-domain
    └─ load scraping-engineering-python skill

7. Caching raw HTML

Lawson's central discipline: store raw HTML, not just parsed records. Re-extraction is free; re-crawling is expensive. Use tools.scraping.DiskCache or SQLiteCache for any non-trivial crawl. Detail in scraping-engineering-python.

8. Parser-performance baseline

Lawson's 1000-iteration benchmark on the same page: regex 5.5 s · lxml 7.0 s · BeautifulSoup pure-Python 42.8 s. Default to BeautifulSoup(html, "lxml") (fast + ergonomic). Only profile for selectolax (~10× faster than lxml on huge pages) when the parser is the actual bottleneck — network usually dominates.

9. Universal anti-patterns

• time.sleep() as a wait strategy in browser code — use wait_for_selector.
• verify=False as a habit — only for self-signed dev hosts.
• Indexing by visual position (soup.find_all('div')[3]) — use semantic selectors.
• Single giant try/except wrapping the whole crawl — swallows real bugs.
• Re-crawling on every run — cache raw HTML.
• Hard-coded selectors with no graceful "MISSING DATA" fallback.
• Parsing HTML with regex (parser preferred; regex only on extracted text).
• Hammering a server with no delay — use Throttle.
• Running a headless browser when a JSON XHR exists.

10. Ship gate (universal)

• Decision tree run; the chosen approach is the cheapest one that works.
• Structured-data shortcuts checked before HTML parsing.
• references/politeness-and-ratelimiting.md rules applied (robots.txt, throttle, identification, backoff).
• Selectors verified against ≥3 records.
• Missing fields normalised to None; one bad record does not crash the crawl.
• Errors surfaced by class (4xx vs 5xx vs 429 vs SoftBlock).
• Raw HTML cached for non-trivial crawls.
• Output stored as Parquet + JSONL with manifest (source, fetched_at, headers, selector_versions, n_rows, n_errors).
• If headless was used, the trigger from Step 5 is documented.

See also

• scraping-engineering-python — caching, concurrency, dynamic content, Scrapy
• dataset-discovery-and-analysis — try this before scraping
• data-quality-assessment — score scraped batches on the four-axis model
• evidence-discipline — when scraping for OSINT/DD, evidence rules override defaults