Paywall Bypass

Paywall Bypass Helper

Identify paywall provider types and generate bypass strategies for web scraping, content extraction, and research across 500+ news sites.

Use this skill when

• The user asks to "bypass paywall", "extract article content", or "get full text" from a paywalled URL
• The user needs to scrape or research content from news sites with subscription walls
• The user asks what paywall technology a specific site uses
• The user needs to write scripts that handle paywalled content programmatically
• The user asks about cookie behavior that triggers or avoids paywalls

Do not use this skill when

• The task involves cracking DRM, decrypting encrypted content, or circumventing hard encryption
• The user requests illegal access to credentials or personal data
• The target is a non-paywall access control (login-only portals, intranets)
• The user wants to distribute full paywalled content commercially

How paywalls work

Paywalls generally work through one of these mechanisms:

1. Metered paywalls — track article views via cookies/localStorage; after N free articles, block access
2. Hard paywalls — content is loaded but hidden behind CSS/JS overlay; blocking the JS reveals text
3. Server-side paywalls — content is not sent to the browser; requires external fetching (archive, cache, bot UA)
4. Hybrid paywalls — combination of client-side blocking + server-side checks

Paywall provider identification

To identify which paywall provider a site uses, inspect the page source for these signatures:

For a complete list of patterns, see references/paywall-patterns.md.

Bypass technique decision tree

Step 1: Identify the paywall type

Check page source for:
  1. Scripts matching known paywall providers (table above)
  2. Cookie names (TDNotesRead, TID_ID, ra, blaize_session, etc.)
  3. AMP version availability (<link rel="amphtml">)
  4. JSON-LD article data (script[type="application/ld+json"])
  5. CSS class patterns (paywall, overlay, modal, gated)

Step 2: Choose bypass technique

Step 3: Implement bypass

For each technique, see references/content-extraction.md for implementation details including code patterns, selectors, and fallback strategies.

Cookie management per provider

Different paywall providers use different cookies to track access:

Important: Some sites require cookies to be preserved (e.g., authentication-based bypass). Check references/cookie-rules.md for per-site rules.

Common media groups

Many news sites share paywall infrastructure. Identifying the media group simplifies bypass:

Implementation guidelines

When writing a scraper/bypass script

1. Always inspect first — check page source for paywall scripts before attempting bypass
2. Start with the least invasive technique — cookie clear > script block > UA spoof > external fetch
3. Handle JavaScript-rendered content — many modern sites use React/Next.js; use Puppeteer/Playwright if needed
4. Respect rate limits — do not hammer servers with requests; add delays between fetches
5. Use headless browsers for JS-rendered paywalls — some paywalls only trigger after JS execution
6. Validate output — verify extracted text is complete, not truncated teaser content

When analyzing a new/unlisted site

1. Check for AMP version: <link rel="amphtml" href="...">
2. Search for paywall scripts: .piano., .tinypass., .poool., .sophi., .evolok., .qiota.
3. Check localStorage keys for meter tracking
4. Try Googlebot UA: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
5. Check for JSON-LD: script[type="application/ld+json"] with articleBody
6. Try archive.is as fallback

References

• references/paywall-patterns.md — Complete regex patterns for all paywall providers
• references/content-extraction.md — JSON-LD, AMP, archive.is, Next.js extraction methods
• references/cookie-rules.md — Per-site cookie management rules

Examples

• examples/identify-paywall.md — Walkthrough for detecting paywall type from HTML source
• examples/bypass-strategy.md — Step-by-step bypass recipe for a specific site