wikimedia-wikitext

name	wikimedia-wikitext
description	Parse, extract, and manipulate Wikipedia and MediaWiki wikitext (wiki markup, templates, infoboxes, citations, links) using proper AST-based tooling instead of fragile regex patterns
license	MIT
compatibility	opencode
depends_on	["wikimedia-api-access"]
skill_discovery_hints	[{"keywords":["wikitext","wiki markup","parse","mwparserfromhell","wikitext parsing","template parsing"]},{"keywords":["wikitext AST","syntax tree","section parsing","link extraction","template expansion"]}]
last_verified	"2026-06-10T00:00:00.000Z"

⚠️ User-Agent required: The API examples below use the Action API and REST API. All requests must include a descriptive User-Agent header or they will be blocked. See the wikimedia-api-access skill for the correct format.

SOP: The Two Strategies

There are two fundamentally different approaches to reading and extracting data from MediaWiki wikitext. Choose based on whether you need to write back wikitext changes.

Strategy	When to use	Tool
AST Parsing	You need to modify wikitext or extract structured data from raw markup	mwparserfromhell
Parsoid HTML	You only need to read page content as structured text/data	MediaWiki REST API (/html endpoint) + BeautifulSoup/lxml

Never mix the two — do not parse Parsoid HTML to reconstruct wikitext, and do not use regex on raw wikitext to extract data.

---

SOP: AST Parsing with mwparserfromhell

Wikitext is non-regular, context-dependent, and recursive (templates inside templates inside image captions). Never use regex to parse wikitext structures like templates, tables, or links. Always use mwparserfromhell, an LL(1)-based parser that builds a proper Abstract Syntax Tree (AST).

Installation

pip install mwparserfromhell

Safe Template Extraction

import mwparserfromhell

wikitext = "[[File:Example.jpg|thumb|{{Location|40|-74}}]] Text with {{Template|param=value|nested={{Value}}}}."
code = mwparserfromhell.parse(wikitext)

for template in code.filter_templates():
    if template.name.matches("Template"):
        param = template.get("param").value
        print(f"Found param: {param}")

Mutating Wikitext

import mwparserfromhell

code = mwparserfromhell.parse(wikitext)
for template in code.filter_templates():
    if template.name.matches("Infobox"):
        template.add("image", "new_image.jpg")

updated_wikitext = str(code)

Stripping Markup to Plain Text

code = mwparserfromhell.parse(wikitext)
plain_text = code.strip_code()  # Removes templates, comments, wikilinks, etc.

⚠️ strip_code() removes:

• HTML comments (<!-- ... -->)
• Templates (replaced with their rendered output if available, else stripped)
• Wiki markup formatting

strip_code() preserves:

• Plain text content
• External URLs

Use str(code) when you need the raw wikitext with all markup intact.

Filtering Specific Node Types

# All templates
code.filter_templates()

# All wikilinks ([[...]])
code.filter_wikilinks()

# All external links ([...])
code.filter_external_links()

# All tags (<ref>, <nowiki>, etc.)
code.filter_tags()

# All headings
code.filter_headings()

# Filter by name pattern
code.filter_templates(matches=lambda name: name.startswith("Infobox"))

Parameter Access Patterns

template = code.filter_templates(matches="Infobox person")[0]

# Safe access with default
name = template.get("name").value
# raises ValueError if param doesn't exist

# Check existence first
if template.has("birth_date"):
    bd = template.get("birth_date").value

# Get all parameters
for param in template.params:
    print(f"{param.name}: {param.value}")

# Add or update a parameter
template.add("new_param", "value")
template.add("existing_param", "new_value")  # Replaces if exists

---

SOP: Parsoid HTML Strategy (Read-Only)

If you do not need to write back wikitext changes, do not parse wikitext at all. Use the MediaWiki Core REST API's /html endpoint, which returns clean, semantic HTML5/RDFa generated by Parsoid (natively embedded in MediaWiki — the standalone JS version is obsolete).

Fetching Page HTML

import requests

url = "https://en.wikipedia.org/w/rest.php/v1/page/Python_(programming_language)/html"
headers = {"User-Agent": "MyTool/1.0 ([e-mail])"}

response = requests.get(url, headers=headers, timeout=30)
if response.status_code == 200:
    html = response.text
    # Parse with BeautifulSoup or lxml

Extracting Tables as DataFrames

Wikitext tables ({| ... |}) contain deep edge cases (implicit row spans, embedded CSS). Never parse them manually.

import pandas as pd
from bs4 import BeautifulSoup

soup = BeautifulSoup(html, "html.parser")
tables = pd.read_html(str(soup))  # pandas handles HTML tables natively

for i, df in enumerate(tables):
    print(f"Table {i}: {df.shape[0]} rows × {df.shape[1]} columns")

Extracting Section Content

soup = BeautifulSoup(html, "html.parser")
for section in soup.find_all(["h1", "h2", "h3", "h4", "h5", "h6"]):
    heading = section.get_text(strip=True)
    # Next siblings until the next heading are this section's content

---

SOP: Anti-Patterns to Avoid

❌ Anti-Pattern	Why It Breaks	✅ Correct Approach
r'\{\{Infobox(.*?)\}\}' for template params	Breaks on nested templates, multi-line params, pipes in wikilinks	mwparserfromhell.filter_templates()
Splitting on `	` to get table cells	Pipes inside wikilinks `[[a
r'\[\[(.*?)\]\]' for wikilinks	Misses piped links, interwiki links, category links, file links	mwparserfromhell.filter_wikilinks()
r'<ref>(.*?)</ref>' for references	Misses self-closing <ref name="x" />, multi-line refs, nested refs	mwparserfromhell.filter_tags()
Manually stripping HTML from Parsoid output	Fragile — Parsoid's HTML structure may change across MediaWiki versions	BeautifulSoup/lxml with semantic selectors

---

SOP: Troubleshooting Edge Cases

Complex Tables

Fetch via /html endpoint and pass to pandas.read_html(). Do not manually chunk {| and |} tokens.

Invisible/Stripped Content

mwparserfromhell.strip_code() removes HTML comments and template markup. Use code.get_sections() if you need to preserve section structure, or str(code) for raw wikitext.

Nested Templates

mwparserfromhell handles arbitrary nesting depth — let the parser do the work:

code = mwparserfromhell.parse("{{Outer|{{Inner|param={{Deepest|value}}}}}}")
for t in code.filter_templates():
    print(t.name)  # Correctly identifies all three templates

Unicode and Special Characters

Wikitext may contain non-ASCII characters, HTML entities (&, <), and magic words. mwparserfromhell handles these correctly; regex approaches will not.

---

Tooling

This skill includes helper scripts, reference docs, and templates:

🔧 Wikitext Inspector (scripts/test-mwparser.sh)

Inspect any wikitext file and get a structured overview of what's in it — templates, wikilinks, tags, headings, sections, and potential issues. Also doubles as an installation smoke test when called with no arguments.

# Smoke test (no args) — verify installation
./scripts/test-mwparser.sh

# Inspect a wikitext file
./scripts/test-mwparser.sh page.wikitext

# Pipe mode — inspect wikitext from stdin
curl -s "https://en.wikipedia.org/w/index.php?title=Python_(programming_language)&action=raw" \
  | ./test-mwparser.sh -

# Usage: ./test-mwparser.sh [file.wikitext | -]
#   (no args)  — run installation smoke test
#   file.wikitext — parse file and report structure
#   -          — read wikitext from stdin (pipe mode)

Reports: total AST nodes, templates with parameter previews, wikilinks with display text, tags, headings (section outline), plain text length, and potential issues (e.g., empty file, very large pages).

🔧 Parsoid HTML Fetcher (scripts/fetch-parsoid-html.sh)

Fetch a Wikipedia page as clean Parsoid HTML and save to a file.

# Fetch page HTML
./scripts/fetch-parsoid-html.sh "Python (programming language)"

# Fetch into a specific output file
./scripts/fetch-parsoid-html.sh "Albert Einstein" albert.html

📚 mwparserfromhell Reference (references/mwparserfromhell-guide.md)

Complete reference covering:

• All node types and filter methods
• Template parameter manipulation
• Section and heading handling
• Tag and comment handling
• Common patterns for infoboxes, citations, lists
• Performance considerations for large pages

📚 Parsoid HTML API Reference (references/parsoid-html-api.md)

Complete guide to the MediaWiki REST API /html endpoint:

• Endpoint structure and parameters
• Content negotiation (JSON vs HTML)
• If-None-Match / ETag caching
• Section-only retrieval
• Handling revisions vs latest
• Error response guide

📚 Wikitext Pitfalls (references/wikitext-pitfalls.md)

Deep reference of documented edge cases:

• Table syntax gotchas (implicit row/colspan, multi-line cells)
• Template argument separation (pipes in different contexts)
• Comment placement affecting template parsing
• Tag extension boundaries (,

  , )

• Unicode normalization issues
• Parser function intricacies
• Behavior of strip_code() with various markup types

🐍 Wikitext Parser Template (assets/parse-wikitext.py)

Ready-to-use Python script with:

• Template extraction and parameter manipulation
• Wikilink and external link extraction
• Section splitting and plain-text conversion
• Infobox data extraction
• Citation list extraction
• Multi-file batch processing

# Extract all templates from a wikitext file
python3 assets/parse-wikitext.py page.wikitext --templates

# Extract all internal links
python3 assets/parse-wikitext.py page.wikitext --links

# Convert to plain text
python3 assets/parse-wikitext.py page.wikitext --plaintext

🐍 Parsoid Extractor Template (assets/parsoid-extractor.py)

Ready-to-use Python script for the HTML DOM strategy:

# Fetch a page and extract tables as CSV
python3 assets/parsoid-extractor.py "Python (programming language)" --tables

# Extract section text
python3 assets/parsoid-extractor.py "Albert Einstein" --sections

# Extract infobox data from rendered HTML
python3 assets/parsoid-extractor.py "Berlin" --infobox

🧩 Table to DataFrame Template (assets/table-extractor.py)

Example script for converting wikitext tables to pandas DataFrames via the Parsoid HTML pipeline, with CSV/JSON export.

---

Cross-References

Related Skill	Why
wikimedia-api-access	User-Agent and API patterns — the foundation all API calls build on
wikipedia-templates	Template syntax — mwparserfromhell is the primary parsing tool
wikipedia-wikitables	Table parsing — always use Parsoid HTML + pandas.read_html(), never regex
wikipedia-citations	Citation/reference extraction from wikitext
wikipedia-page-anatomy	Page structure — sections, infoboxes, navboxes parsed from wikitext
pywikibot	Bot framework — uses mwparserfromhell for template manipulation
wiktionary-and-wikisource	Dictionary/text extraction patterns using AST-based parsing
wikivoyage	Wikivoyage listing templates ({{see}}, {{eat}}, etc.) are structured wikitext templates parsed with mwparserfromhell