Download full-history PBFs → osm_versions.parquet + osm_changes.parquet
python scripts/osm_data/download_history.py
Runs osmium tags-filter --omit-referenced then osmium time-filter, then pyosmium streams results. Iterates the 4 extracts (us, pr, usvi, american_oceania) and applies N-way iterative (type, id) dedup before writing the final concatenated parquets. Controlled by download.osm.* in config.yaml.
Advance download.osm.end_date before a refit to the current cutoff (e.g. the Overture release date) so the fit sees the latest edits — it is the osmium time-filter upper bound, and a stale value silently caps the observation window (start_date stays 2016-01-01). This is a standing monthly step; see skills/full-data-pull step 1.
Gotcha — extract order matters for performance, not correctness. The dedup keeps a seen_type_ids set that grows monotonically across extracts: the first extract is stream-copied as-is, each subsequent extract is filtered against the union of all prior extracts. Put the largest extract first (us is already first in scripts/osm_data/download_history.py). Reordering to put a small territory first forces the huge us extract to be filtered against a tiny seen-set — same correctness, much more wasted work.
Gotcha — _concat_history memory at nationwide scale. The seen_type_ids set holds one (str, int) tuple per element version across all extracts. Python sets of small tuples are ~150 bytes each → at ~50M US versions, the set alone can reach ~7 GB RSS. This isn't a regression from the pre-territory pairwise (US, PR) version (pass-1 already grew the set during US stream-copy), but it's worth watching peak memory during the final concat step. If the box OOMs, the fix is to spill the seen set to a disk-backed (type, id) store (e.g. DuckDB or RocksDB) rather than the in-memory set.
Gotcha — per-extract 404 tolerance: if Geofabrik ever stops publishing a territory's *-internal.osh.pbf, the loader logs a warning, skips that extract, and continues. The territory's POIs still flow through downstream stages but the rater falls back to the global-mean δ for its shared_labels. Pre-flight is cheap — see docs/data-sources.md for the cookie/URL check.
