| name | paywall-bypass |
| description | Identify paywall types and generate bypass strategies for web scraping and content extraction. Covers 500+ news sites, major paywall providers (Piano, TinyPass, Poool, etc.), and extraction techniques (bot spoofing, AMP, JSON-LD, archive fetching). |
| version | 1.0.0 |
| author | Extracted from Bypass Paywalls Clean (magnolia1234) |
| date | "2026-04-04T00:00:00.000Z" |
Paywall Bypass Helper
Identify paywall provider types and generate bypass strategies for web scraping, content extraction, and research across 500+ news sites.
Use this skill when
- The user asks to "bypass paywall", "extract article content", or "get full text" from a paywalled URL
- The user needs to scrape or research content from news sites with subscription walls
- The user asks what paywall technology a specific site uses
- The user needs to write scripts that handle paywalled content programmatically
- The user asks about cookie behavior that triggers or avoids paywalls
Do not use this skill when
- The task involves cracking DRM, decrypting encrypted content, or circumventing hard encryption
- The user requests illegal access to credentials or personal data
- The target is a non-paywall access control (login-only portals, intranets)
- The user wants to distribute full paywalled content commercially
How paywalls work
Paywalls generally work through one of these mechanisms:
- Metered paywalls — track article views via cookies/localStorage; after N free articles, block access
- Hard paywalls — content is loaded but hidden behind CSS/JS overlay; blocking the JS reveals text
- Server-side paywalls — content is not sent to the browser; requires external fetching (archive, cache, bot UA)
- Hybrid paywalls — combination of client-side blocking + server-side checks
Paywall provider identification
To identify which paywall provider a site uses, inspect the page source for these signatures:
| Provider | Detection Script/Pattern | Sites Using |
|---|
| Piano.io | .piano.io/, .piano.io/xbuilder/ | Foreign Policy, DN.no, Funke, Vocento, Berlingske |
| TinyPass (Piano) | .tinypass.com/, js.tinypass.com | Chicago Sun-Times, Fortune, Adweek, Corriere.it, Digiday |
| Poool.fr | .poool.fr/ | Alternatives Economiques, Challenges, Elle.fr, Le Télégramme, GCF group |
| Sophi.io | .sophi.io/ | Adweek, Automobilwoche, Business Insider, Crain Comm |
| Evolok | .evolok.net/ | El País, Vocento group, Leaky Paywall |
| Qiota | .qiota.com/ | AutoPlus.fr, Cosmopolitan.fr, Nice-Matin group, Rossel group |
| Pelcro | js.pelcro.com/ | Domani, Foreign Affairs, Crain Comm |
| Cxense | .cxense.com/ | Barron's, Business Insider JP, IPM group |
| Blueconic | .blueconic.net/ | Bridge Tower Media |
| Memberstack | [data-memberstack-app] | Various SaaS-gated sites |
| Steady | steadyhq.com/ | American Purpose, some Ghost sites |
| Leaky Paywall | /leaky-paywall/ | WordPress sites with Leaky plugin |
| Wallkit | wallkit.net | WordPress subscription sites |
| AMP Access | .ampproject.org/v0/amp-access-, .ampproject.org/v0/amp-subscriptions- | Many sites with AMP versions |
For a complete list of patterns, see references/paywall-patterns.md.
Bypass technique decision tree
Step 1: Identify the paywall type
Check page source for:
1. Scripts matching known paywall providers (table above)
2. Cookie names (TDNotesRead, TID_ID, ra, blaize_session, etc.)
3. AMP version availability (<link rel="amphtml">)
4. JSON-LD article data (script[type="application/ld+json"])
5. CSS class patterns (paywall, overlay, modal, gated)
Step 2: Choose bypass technique
| Technique | When to use | How |
|---|
| Block paywall script | Script-based paywall (TinyPass, Piano, Poool, etc.) | Block the script URL; content loads but paywall overlay doesn't trigger |
| Clear cookies | Metered paywall tracking views | Remove tracking cookies; localStorage.clear() + sessionStorage.clear() |
| Bot UA spoofing | Server serves full content to crawlers | Set User-Agent to Googlebot/Bingbot; add Referer: https://www.google.com/ |
| AMP page | Site has AMP version with relaxed paywall | Navigate to AMP URL or unhide content blocked by amp-access/amp-subscriptions |
| JSON-LD extraction | Article body in structured data | Parse script[type="application/ld+json"] for articleBody key |
| Archive fetch | Server-side paywall, no client-side text | Fetch article from archive.is or Google cache |
| Next.js data | Sites using Next.js SSR | Extract from script#__NEXT_DATA__ |
| External link | No bypass possible client-side | Generate link to archive.is or Google Search Tool for manual fetch |
Step 3: Implement bypass
For each technique, see references/content-extraction.md for implementation details including code patterns, selectors, and fallback strategies.
Cookie management per provider
Different paywall providers use different cookies to track access:
| Provider | Tracking Cookie(s) | Action |
|---|
| Piano (general) | None specific | Block script, no cookie action needed |
| TinyPass (Piano) | xbc (HBR) | Drop specific cookie |
| DPG Media (NL) | TID_ID | Drop this cookie to reset meter |
| Haaretz Group | ra | Drop this cookie |
| Adweek/Zephr | blaize_session | Drop this cookie |
| Pitchfork | pay_ent_msmp | Drop this cookie |
| Business Standard | userUid | Drop this cookie |
| eNotes | ENOTESID | Drop this cookie |
| DN Media (no) | AnonUserCookie | Drop this cookie |
Important: Some sites require cookies to be preserved (e.g., authentication-based bypass). Check references/cookie-rules.md for per-site rules.
Common media groups
Many news sites share paywall infrastructure. Identifying the media group simplifies bypass:
| Group | Key Sites | Paywall Type |
|---|
| News Corp Australia | theaustralian.com.au, heraldsun.com.au, dailytelegraph.com.au | AMP subscriptions + hard paywall |
| McClatchy (USA) | bnd.com, miamiherald.com, sacbee.com, kansascity.com | Piano + Googlebot |
| Gannett | azcentral.com, freep.com, indystar.com | Googlebot UA |
| Hearst (USA) | houstonchronicle.com, sfchronicle.com, statesman.com | Custom script blocking |
| Lee Enterprises | townnews.com-based sites | TownNews/Leaky paywall |
| Vocento (ES) | abc.es, elmundo.es (not), larioja.com, ideal.es | Evolok + AMP access |
| GEDI (IT) | repubblica.it, lastampa.it, huffingtonpost.it | Piano + Googlebot |
| DPG Media (NL) | volkskrant.nl, trouw.nl, demorgen.be | Custom temptation.js |
| Groupe Rossel (FR/BE) | lavoixdunord.fr, lesoir.be, sudinfo.be | Qiota |
| Funke (DE) | abendblatt.de, morgenpost.de, waz.de | Piano |
| Crain Comm (USA) | adage.com, autonews.com, chicagobusiness.com | Pelcro + Sophi |
| DN Media (NO) | dn.no, rechargenews.com, upstreamonline.com | Custom + fetch from archive |
| Conde Nast (USA) | newyorker.com, vogue.com, wired.com, gq.com | Custom + script blocking |
| Advance Local (USA) | nj.com, mlive.com, cleveland.com, al.com | Sophi + DOMPurify |
Implementation guidelines
When writing a scraper/bypass script
- Always inspect first — check page source for paywall scripts before attempting bypass
- Start with the least invasive technique — cookie clear > script block > UA spoof > external fetch
- Handle JavaScript-rendered content — many modern sites use React/Next.js; use Puppeteer/Playwright if needed
- Respect rate limits — do not hammer servers with requests; add delays between fetches
- Use headless browsers for JS-rendered paywalls — some paywalls only trigger after JS execution
- Validate output — verify extracted text is complete, not truncated teaser content
When analyzing a new/unlisted site
- Check for AMP version:
<link rel="amphtml" href="...">
- Search for paywall scripts:
.piano., .tinypass., .poool., .sophi., .evolok., .qiota.
- Check localStorage keys for meter tracking
- Try Googlebot UA:
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
- Check for JSON-LD:
script[type="application/ld+json"] with articleBody
- Try archive.is as fallback
References
references/paywall-patterns.md — Complete regex patterns for all paywall providersreferences/content-extraction.md — JSON-LD, AMP, archive.is, Next.js extraction methodsreferences/cookie-rules.md — Per-site cookie management rules
Examples
examples/identify-paywall.md — Walkthrough for detecting paywall type from HTML sourceexamples/bypass-strategy.md — Step-by-step bypass recipe for a specific site
