wikimedia-api-access

name	wikimedia-api-access
description	Access Wikipedia and Wikimedia APIs (REST, Action API, SPARQL) with correct User-Agent headers, rate limiting, and 429/403 error handling
license	MIT
compatibility	opencode
skill_discovery_hints	[{"keywords":["Wikipedia API","Action API","REST API","API endpoint","api.php","rest_v1","User-Agent","rate limit","API call"]},{"keywords":["Site Matrix","sitematrix","domain mapping","language code","language domain","yue wikipedia","zh-yue","interlanguage"]},{"keywords":["page summary","page extract","extintro","exintro","page content","fetch article","get page"]}]
last_verified	"2026-06-10T00:00:00.000Z"

All requests to Wikimedia APIs must include a descriptive User-Agent header or they will be blocked (HTTP 403 or 429). This is enforced by the Wikimedia Foundation User-Agent Policy.

💡 Pi users: If you use the pi coding agent, install the companion extension from this repo to automatically inject the User-Agent header on every curl, wget, python, and node command that targets a Wikimedia server — no manual intervention needed. See the README for setup instructions.

💡 Not sure which tool to use? See wikimedia-api-strategy for a decision framework comparing REST API, Action API, SPARQL, SQL replicas, EventStreams, and Pywikibot — with latency/complexity/authentication trade-offs and common anti-patterns.

User-Agent Format

The generic format is:

<client name>/<version> (<contact information>) <project identifier>/<version>

Parts that are not applicable can be omitted. The contact information should be an email address, website URL, or wiki user page.

Required Pattern

headers = {
    'User-Agent': 'YourBot/1.0 (https://your-site.com; [e-mail]) ProjectName'
}

The ProjectName (e.g. ContentGapResearch) helps Wikimedia distinguish traffic from different tools.

Concrete Example

import requests

headers = {
    'User-Agent': 'my-wiki-bot/1.0 (https://example.com; [e-mail]) ContentGapResearch'
}

response = requests.get('https://en.wikipedia.org/w/api.php', headers=headers, timeout=30)

Why This Matters

• Requests without a User-Agent or with generic agents (python-requests/x, curl, Python-urllib) are blocked with HTTP 403 or 429 errors.
• A descriptive UA allows Wikimedia to contact you if your bot misbehaves.
• Always include "bot" in the UA string (case-insensitive) for automated agents — this helps Wikimedia classify traffic correctly and provide accurate statistics.
• Never copy a browser's UA string for bot requests — bot-like behavior with a browser UA will be treated as malicious.

Key API Endpoints

API	Base URL	Common Use
Action API	https://en.wikipedia.org/w/api.php	Search, page content, edits, categories
REST API	https://en.wikipedia.org/api/rest_v1/	Page summaries, mobile content, transforms
Pageviews	https://wikimedia.org/api/rest_v1/metrics/pageviews/	Traffic statistics
Commons Analytics	https://wikimedia.org/api/rest_v1/metrics/commons-analytics/	GLAM category/file usage stats (monthly, pre-compiled)
Lift Wing ML	https://api.wikimedia.org/service/lw/inference/v1/models/	ML predictions (revert risk, article quality, topics)
Site Matrix	https://en.wikipedia.org/w/api.php?action=sitematrix	Get correct domains for all Wikimedia wikis (handles code/domain mismatches like yue→zh-yue)
Wikidata Query (SPARQL)	https://query.wikidata.org/sparql	Structured data queries
Toolforge	https://*.toolforge.org/	Community tool hosting

💡 These APIs are project-agnostic. The Action API and REST API shown above with en.wikipedia.org are MediaWiki APIs that work on any Wikimedia project — just swap the domain. For example:

• Commons: https://commons.wikimedia.org/w/api.php — search media files, fetch EXIF metadata, inspect categories
• Wikidata: https://www.wikidata.org/w/api.php — query entities, manage labels and descriptions, fetch claims
• Wiktionary: https://en.wiktionary.org/w/api.php — dictionary data, word definitions, etymology
• Meta-Wiki: https://meta.wikimedia.org/w/api.php — global user info, cross-wiki settings, Wikimedia Foundation policies

The query parameters and response structures are the same; only the domain and the wiki's content differ.

General Implementation Pattern

import requests
import time

SESSION = requests.Session()
SESSION.headers.update({
    'User-Agent': 'my-wiki-bot/1.0 (https://example.com; [e-mail]) ContentGapResearch'
})

def wikimedia_request(url, params=None, max_retries=3):
    for attempt in range(max_retries):
        resp = SESSION.get(url, params=params, timeout=30)
        if resp.status_code == 200:
            return resp.json()
        if resp.status_code == 429:
            retry_after = int(resp.headers.get('Retry-After', 10))
            time.sleep(retry_after)
            continue
        if resp.status_code == 403:
            raise PermissionError(
                "403 Forbidden — likely a missing or invalid User-Agent. "
                "See: https://foundation.wikimedia.org/wiki/Policy:Wikimedia_Foundation_User-Agent_Policy"
            )
        resp.raise_for_status()
    raise Exception(f"Request failed after {max_retries} retries")

# Typical response structure (Action API query):
# {
#   "batchcomplete": "",
#   "query": {
#     "pages": {
#       "12345": {                                # ← page ID as string key
#         "pageid": 12345,
#         "ns": 0,
#         "title": "Albert Einstein",
#         ...other requested properties...
#       }
#     }
#   },
#   "continue": {"continue": "-||", ...}        # ← pagination cursor (if more results)
# }
# Access: data["query"]["pages"]["12345"]["title"]
# Iterate: for page_id, page_data in data["query"]["pages"].items():

Browser-Based JavaScript

Browser JavaScript cannot control the User-Agent header (the browser sets it). Instead, use the Api-User-Agent header:

fetch('https://en.wikipedia.org/w/api.php?action=query&format=json', {
    headers: new Headers({
        'Api-User-Agent': 'MyScript/1.0'
    })
})

Rate Limiting & Error Handling Guardrails

1. Connection reuse — Always use a requests.Session() (or equivalent) to reuse connections. Do not create a new connection per request.
2. Retry-After — On 429, respect the Retry-After header value. Never retry immediately.
3. Pacing & Batching — For batch operations, add a small delay (at least 0.5s) between requests. For the Action API, respect the maxlag parameter. Always use the largest batch size the API supports (e.g., rvlimit=500, uclimit=500) rather than fetching items one at a time. See the SOP: Batching and Pagination for Efficiency in the wikipedia-edit-history skill for detailed patterns.
4. 403 handling — A 403 almost always means a bad/missing UA. Check the UA string before debugging anything else.
5. User-Agent per project — Parameterize the contact info so users can swap in their own details. Never hardcode someone else's email.
6. SPARQL queries — For Wikidata Query Service, always set the UA and use &format=json. Consider using SPARQLWrapper with the agent parameter.

429 Retry-After Handling

When a 429 (Too Many Requests) response is received, Wikimedia includes a Retry-After header specifying the number of seconds to wait. Do not retry immediately with a fixed backoff (e.g., always waiting exactly 5 seconds) — this is counterproductive and may lead to a temporary ban. Always use the server-supplied value:

if resp.status_code == 429:
    retry_after = int(resp.headers.get('Retry-After', 10))
    time.sleep(retry_after)
    continue  # retry the request in a loop

Caching Strategy (Prevents Redundant Calls)

When fetching the same set of pages across multiple runs (e.g., re-running a tool with different parameters), a simple disk cache avoids redundant HTTP requests and reduces rate-limit pressure.

Recommended pattern — JSON key-value cache:

import json, os

CACHE_FILE = ".api_cache.json"
_cache = {}

if os.path.exists(CACHE_FILE):
    with open(CACHE_FILE) as f:
        _cache = json.load(f)

def fetch_with_cache(lang, title):
    key = f"{lang}:{title}"
    if key in _cache:
        return _cache[key]
    url = f"https://{lang}.wikipedia.org/api/rest_v1/page/summary/{title}"
    resp = requests.get(url, headers=HEADERS, timeout=15)
    if resp.status_code == 200:
        data = resp.json()
        result = data.get("extract", "")
        _cache[key] = result
        with open(CACHE_FILE, "w") as f:
            json.dump(_cache, f)
        return result
    return None

Key points:

• Use {lang}:{title} as the cache key to avoid collisions
• Write back to disk after each fetch (appending is fine for ~300 entries)
• Keep cache files gitignored (they're session artifacts)
• For large caches (>1000 entries), consider SQLite instead of JSON

Structured Error Returns from Fetch Functions

When a fetch fails, return a dict with lead and error fields rather than a bare None. This lets callers distinguish between different failure modes without making additional API calls:

def fetch_lead(lang, title) -> dict:
    """Returns {"lead": str|None, "error": str|None}."""
    url = f"https://{lang}.wikipedia.org/api/rest_v1/page/summary/{title}"
    resp = requests.get(url, timeout=15)
    if resp.status_code == 200:
        data = resp.json()
        extract = data.get("extract", "")
        if extract and extract.strip():
            return {"lead": extract, "error": None}
        # HTTP 200 but empty extract — investigate why
        page_type = data.get("type", "")
        if page_type == "disambiguation":
            return {"lead": None, "error": "Disambiguation page (no lead text)"}
        if data.get("description"):
            return {"lead": None, "error": f"Has description but no extract; type={page_type}"}
        return {"lead": None, "error": f"No content; type={page_type}"}
    return {"lead": None, "error": f"HTTP {resp.status_code}"}

This pattern lets consumers show actionable error messages:

❌  scn    Has description but no extract; type=standard
❌  mhr    No content; type=standard
❌  nb     HTTP 500

Common Causes of 429 Responses

• Too many requests in a short window even with a proper User-Agent. Stay under ~2 requests/second for batch operations, and add deliberate delays (0.3–0.5s) between requests.
• Too many titles in a single titles parameter. Keep batches under 50 titles per call for prop=pageprops and 50 IDs per call for wbgetentities.
• Fetching very large pages via action=parse without a section limit. Add rvsection=0 or section=0 if you only need the lead section, or use exintro with prop=extracts for a concise summary.

⚠️ Login Username Quirk: Spaces → Underscores

The MediaWiki action=login API requires the internal database form of usernames, where spaces are stored as underscores (_). The lgname parameter will fail silently (returning "Unknown error") if you pass a username with literal spaces, even though that's how the username displays on wiki pages and in the Special:BotPasswords confirmation message.

Always normalize usernames before passing them to lgname:

# ✗ This will FAIL with "Unknown error"
lgname = "AL Wiki MIT"

# ✓ This works
lgname = "AL_Wiki_MIT"

If you're using bot passwords, the same rule applies:

# ✗ Fails
lgname = "AL Wiki MIT@mybot"

# ✓ Works
lgname = "AL_Wiki_MIT@mybot"

Note: action=login also requires POST, not GET. See the MediaWiki API documentation for the full login flow (including the two-step NeedToken case).

🛡️ Critical: always verify login succeeded. A login failure (wrong password, expired token, network issue) returns NeedToken rather than raising an error. If you proceed without checking, you'll get an anonymous CSRF token (+\) and your edits will be attributed to a temporary account (~2026-XXXXX-XX) instead of your bot. Always:

1. Assert login_resp["login"]["result"] == "Success"
2. Verify userinfo matches the expected username
3. Use assert="user" on every write request

See the wikimedia-auth-oauth skill for the full canonical login function with all guardrails built in, and the pywikibot skill's Troubleshooting section for bot password authentication.

Example Use Cases

• Fetch article content: "Get the wikitext for 'Python (programming language)' using the Action API."
• Search: "Search English Wikipedia for articles about 'machine learning' and return the top 10 titles."
• Get page metadata: "Using the REST API, get the description and thumbnail for 'Albert Einstein'."
• SPARQL query: "Query Wikidata for all museums in Paris with their coordinates."
• Batch lookup: "For each page in this list of 50 titles, fetch the page ID and word count from the API."
• Troubleshoot 403: "Check the User-Agent header and confirm it follows the Wikimedia format."

REST API /page/summary Response Details

The /{lang}.wikipedia.org/api/rest_v1/page/summary/{title} endpoint returns a JSON response with several useful fields beyond extract and thumbnail:

{
  "type": "standard",                // ← page type: "standard" or "disambiguation"
  "title": "Albert Einstein",
  "displaytitle": "Albert Einstein",
  "extract": "Albert Einstein was a German-born...",  // ← lead paragraph (may be empty for DAB pages)
  "description": "German-born theoretical physicist",
  "pageid": 736,
  "content_urls": {
    "desktop": {"page": "https://en.wikipedia.org/wiki/Albert_Einstein"},
    "mobile": {"page": "https://en.m.wikipedia.org/wiki/Albert_Einstein"}
  }
}

Key notes:

• Mobile domains deprecated (Oct 2025): Wikimedia unified mobile and desktop domains. en.m.wikipedia.org no longer serves content — it redirects to en.wikipedia.org. Do not construct https://{lang}.m.wikipedia.org/ URLs in new code. The content_urls.mobile.page field returned by the REST API still exists but redirects to the desktop domain. See Unifying mobile and desktop domains.

  To request mobile-formatted content, use one of:

  • URL parameter: https://en.wikipedia.org/wiki/Chess?useformat=mobile (recommended — works on all page views)
  • index.php: https://en.wikipedia.org/w/index.php?title=Chess&mobileformat=1 (legacy, being phased out)
  • Action API parse: action=parse&page=Chess&mobileformat=1 or the newer action=parse&page=Chess&prop=text&useformat=mobile
  • REST API: /page/mobile-html/{title} returns mobile-optimized HTML (stable, not deprecated)

• type field: Use type == "disambiguation" to detect disambiguation pages without parsing templates or categories (more reliable than checking for {{disambiguation}} templates).

• extract may be empty even when the page exists. Disambiguation pages and pages consisting only of an infobox return an empty string for extract but still return HTTP 200.

• Unicode control characters: Some language editions include bare Unicode formatting characters (e.g., U+200E LEFT-TO-RIGHT MARK) in the extract text. These are not removed by .strip() — filter them with ''.join(c for c in text if c.isprintable()) before processing.

• The type field is only available in the REST API, not in the Action API's prop=extracts. If you use the Action API, you'd need to check categories or templates instead.

---

Tooling

This skill includes helper scripts, reference docs, and templates:

🔧 Connectivity Test (scripts/test-api.sh)

Tests 14 endpoints across 6 API families with your User-Agent and reports which ones work.

# Test with the default User-Agent
./scripts/test-api.sh

# Test with YOUR User-Agent
./scripts/test-api.sh "MyBot/1.0 (https://example.com; [e-mail]) MyProject"

Tests 14 endpoints across 6 API families:

• 📡 Core APIs (Action API: siteinfo, search, page content) — 3 tests
• 🌐 REST APIs (page summary, mobile-html) — 3 tests
• 🔗 SPARQL / Wikidata — 2 tests
• 📊 Pageviews API — 1 test
• 🖼️ Commons Analytics API — 3 tests (category metrics, top wikis, top edited)
• 🧠 Lift Wing ML API — 2 tests (POST-based, revert risk + article quality)

Note on Commons Analytics: Data is only available for categories on the allow list and their subcategories. The category-metrics-snapshot and top-wikis-per-category tests use Smithsonian_American_Art_Museum (an allow-listed category) and should return data. See references/endpoints.md for details on the allow list.

📚 API Endpoint Reference (references/endpoints.md)

Full catalog of Wikimedia endpoints organized by API family:

• Action API parameters and pagination pattern
• REST API and RESTBase endpoints
• Pageviews API parameters and date formats
• SPARQL query patterns
• Quick selection guide (task → best endpoint)

Read it when you need to find the right endpoint:

# Load by asking the agent, or reference directly
cat references/endpoints.md

🐍 Python Client Template (assets/user-agent-template.py)

A ready-to-use Python client with:

• Proper User-Agent configuration
• requests.Session with connection reuse
• Rate limiting (configurable delay between requests)
• Automatic retry with exponential backoff on 429/timeouts
• Helpful error messages for 403 (bad UA) and 404
• Convenience methods for all major API types
• Works as a standalone demo script

# Copy and customize
cp assets/user-agent-template.py my_bot.py
# Edit my_bot.py with your User-Agent, then run
python3 my_bot.py

Supports:

from user_agent_template import WikimediaClient

with WikimediaClient("MyBot/1.0 ([e-mail]) MyProject") as client:
    # Action API
    results = client.search("Albert Einstein", limit=5)
    extract = client.get_page_extract("Python (programming language)")
    
    # REST API
    summary = client.page_summary("Albert Einstein")
    
    # SPARQL
    entities = client.sparql_query("SELECT ?item WHERE { ... }")
    entity = client.get_entity("Q937")
    
    # Pageviews
    top = client.top_pageviews("en.wikipedia")

---

CORS: Browser Applications and Cross-Origin Requests

When building browser-based applications (JavaScript, not server-side Python), you may encounter CORS (Cross-Origin Resource Sharing) restrictions. This section explains when this happens and how to handle it.

When CORS Becomes an Issue

Most Wikimedia APIs (Action API, REST API, SPARQL) do send CORS headers and work fine from browser JavaScript. However, media files served from upload.wikimedia.org do not — and this is where you'll encounter CORS errors.

Resource	CORS Headers?	Browser Access
Action API (/w/api.php)	Yes	Works from any origin
REST API (/api/rest_v1/)	Yes	Works from any origin
SPARQL (query.wikidata.org/sparql)	Yes	Works from any origin
Media files (upload.wikimedia.org)	No	Blocked for Canvas/WebGL/background-image
File page redirects (Special:FilePath/)	Yes	Works for <img> display

The Problem: upload.wikimedia.org Has No CORS Headers

Access to image at 'https://upload.wikimedia.org/wikipedia/commons/thumb/...'
from origin 'http://localhost:3000' has been blocked by CORS policy:
No 'Access-Control-Allow-Origin' header is present on the requested resource.

Chrome DevTools shows: net::ERR_BLOCKED_BY_ORB (Opaque Response Blocking).

This affects:

• Canvas pixel access (canvas.getContext('2d').getImageData())
• WebGL textures (texImage2D)
• CSS background-image in some browsers
• Audio/video elements for certain operations

Solution: Proxy Through Your Server

For browser apps that need full access to media content, proxy the request through your own server:

Browser -> Your Server -> Commons API -> Your Server -> Browser
         (same-origin)   (no CORS)     (same-origin)

The browser sees the response as same-origin, so CORS restrictions don't apply.

Server-side pattern (Node.js):

// API endpoint: /api/media?url=...
app.get('/api/media', async (req, res) => {
  const mediaUrl = req.query.url;
  const resp = await fetch(mediaUrl);
  const buffer = await resp.arrayBuffer();
  
  // Cache with SHA-256 hash filename
  const hash = createHash('sha256').update(buffer).digest('hex');
  const cachedPath = `cache/${hash}.jpg`;
  await writeFile(cachedPath, buffer);
  
  res.json({ url: `/cache/${hash}.jpg` });
});

Client-side usage:

// Direct Commons URL - CORS blocked
const directUrl = 'https://upload.wikimedia.org/wikipedia/commons/thumb/...';

// Proxied URL - same-origin, no CORS issues
const proxiedUrl = '/cache/abc123.jpg';
document.getElementById('img').src = proxiedUrl;  // Works!

When You Don't Need a Proxy

• Server-side code (Python, Node.js) — no CORS restrictions, fetch directly
• Simple <img> display — use Special:FilePath/ redirect URLs
• Metadata-only APIs — Action API, REST API, SPARQL all work from browsers

Related Documentation

For detailed thumbnail/media CORS handling, see wikimedia-commons-thumbnails Section 10.

---

Cross-References

Related Skill	Why
wikimedia-api-strategy	Decision framework for choosing the right API
wikimedia-auth-oauth	OAuth 1.0a/2.0 and bot passwords for authenticated requests
wikipedia-error-handling	Retry strategies and backoff patterns for 403/429/5xx responses
wikimedia-commons-thumbnails	CORS proxy patterns for serving Commons media in browser apps
pywikibot	Python bot framework with built-in User-Agent and rate limiting

---