| name | fetch-sitemap |
| description | Fetch, parse and recurse XML sitemaps (the Sitemaps 0.9 protocol — sitemap.xml / sitemapindex.xml) for the UK Parliament web presence, enumerate every public page URL (~1.09M across the estate), and track updates via the RSS/Atom feeds. Use when you need the inventory of public web pages rather than structured API records — especially the many pages with NO JSON/REST API (corporate "About"/living-heritage, visiting, get-involved, site-information, Library landing pages, guidance), to know what the wrapped Members/Bills/Hansard/SPARQL facilities do NOT cover. Entry: www.parliament.uk/sitemapindex.xml; publications via www.publications.parliament.uk/sitemap_index.xml (21 child sitemaps, ~1.04M URLs). A cached snapshot, a concrete 918-feed catalogue, a liveness survey, a D3 web-estate map (/kgx/sitemap-tree) and a feed reader (/kgx/feeds) are all committed — see docs/data-state.md. Parses offline from a saved file too. Generic — works on any site's sitemap. |
fetch-sitemap — enumerate Parliament's public web URLs
Most of this repo wraps Parliament's APIs (structured records). This
facility wraps the sitemaps — the published list of public web page
URLs. That matters because the APIs do not cover every public URL: large
swathes of www.parliament.uk (living-heritage / "About", visiting,
get-involved, site-information, guidance, Library landing pages) are web-only,
with no JSON/REST representation. A sitemap is the authoritative way to
inventory them.
Entry points & scale (~1.09M URLs)
The whole reachable estate is 1,088,193 URLs across 63 sitemaps (probed
2026-06-05). Roots:
https://www.parliament.uk/sitemapindex.xml → 6 child sitemaps = 30,915 URLs
(sitemap.xml 15,966 · business 9,552 · about 3,798 · site-info 1,047 ·
visiting 293 · get-involved 259).http://www.publications.parliament.uk/sitemap_index.xml → 21 child
sitemaps = 1,041,256 URLs. Use the index, not the bare sitemap.xml,
which is only the first child capped at the 50k protocol limit.commonslibrary. (index → 20, ~13k) and lordslibrary. (index → 10, ~2.9k)
are WordPress; members. 67; hansard. a 15-URL stub.bills./committees./questions-statements. have no sitemap — that
content is API-only.
The authoritative location for any host is the Sitemap: line in its
robots.txt (readable via Node fetch). parl sitemap hosts lists entry points.
CLI
parl sitemap hosts
parl sitemap fetch <url>
parl sitemap enumerate <url> [--max-sitemaps N] [--limit N] [--delay-ms 300]
parl sitemap parse-file <path>
enumerate recurses an index into its child sitemaps and returns a flat list
of every <loc> (with lastmod). It is sequential and polite — pass
--delay-ms to space out child fetches, and --limit to cap the URL count.
parl sitemap enumerate https://www.parliament.uk/sitemapindex.xml --delay-ms 300 --limit 50
parl sitemap enumerate http://www.publications.parliament.uk/sitemap_index.xml --limit 50
Cloudflare reality (read this)
Parliament's web hosts sit behind a Cloudflare managed challenge. Whether a
client gets through is transport-fingerprint dependent, not UA dependent
(probed 2026-06-05):
| Client | Result on the sitemap URLs |
|---|
curl (any UA) | 403 "Just a moment…" |
| headless Chrome | 403 (never solves the challenge) |
Node fetch (undici) — what parl uses | passes — returns the real XML |
So parl sitemap fetch/enumerate works live today from Node. We send an
honest identifying User-Agent and do not spoof a browser or solve the
challenge — a sitemap is published for crawlers, so this is a plain polite
GET, not bot-evasion. But the edge behaviour can change without notice, so the
parser is decoupled from the fetch: if a live fetch starts returning 403, open
the sitemap in a normal browser (it passes the challenge in one click), save
the XML, and run parl sitemap parse-file saved.xml. The *-api.parliament.uk
hosts are not challenged at all.
Why this is useful: URLs with no API
Cross-reference the enumerated URLs against the wrapped facilities to find the
public pages that have no API representation (the original motivation):
parl sitemap enumerate … → group by path section → anything under
/about/, /visiting/, /get-involved/, /site-information/,
commonslibrary./lordslibrary. landing pages, etc. is web-only. The library
exposes byHost(urls) for a quick host breakdown.
Cached state, feeds & reader
The whole estate is cached and committed so you don't need to re-crawl — see
docs/data-state.md for the full map. Headlines:
third_party/data/parliament-sitemap/ — raw/*.xml.gz (every sitemap as
fetched), urls.jsonl.gz (all 1.09M <loc>), manifest.json, hierarchy.json.
Built by scripts/cache-parliament-sitemaps.mjs.- Feeds for update-tracking —
feeds.json / feeds.ttl: a concrete
catalogue of 918 RSS/Atom feeds (Commons/Lords Library per
topic/type/tag/author + Bills API) with metadata (host, chamber, scope, group,
tags, facet counts). Built by scripts/build-parliament-feeds.mjs from the
cached taxonomy sitemaps. Sitemaps give the full URL set; feeds give the deltas.
- Liveness —
feeds-liveness.json + feeds-items.json: of 918 feeds, all
return 200 and 917 have items, but only ~419 are active in the last year
(the author feeds are mostly dormant). Built by scripts/probe-feed-liveness.mjs,
refreshed nightly by .github/workflows/refresh-feeds.yml (03:17 UTC).
- UIs on fpkg:
/kgx/sitemap-tree (D3 icicle/treemap/sunburst of the estate)
and /kgx/feeds (a search/filter feed reader over the 419 live feeds).
See reference.md for the data shapes and the full host list.
