| name | conflate-snapshots |
| description | Use when the user wants to match rated OSM POIs with Overture POIs into a unified dataset, partition it for web consumption, and push to Source Cooperative. Triggers: "run conflation", "publish new data", "push new conflated data to Source Cooperative", "bump conflation version", "reconflate with new parameters", "re-upload the partitioned parquet". |
Conflate snapshots + publish to Source Cooperative
Taxonomy-aware matching between rated OSM and Overture, then partition and
upload for web consumption.
Prerequisites
- Rated OSM snapshot (
osm_snapshot_rated.parquet) at versions.snapshot_osm — produced by skills/full-data-pull step 3.
- Overture snapshot (
overture_snapshot.parquet) at versions.snapshot_overture.
- OSM history parquets (
osm_versions.parquet, osm_changes.parquet) at versions.osm_data — regenerated each month by skills/full-data-pull step 2 (via scripts/osm_data/download_history.py). The full re-fit pipeline at skills/model-history-pipeline is only invoked when re-fitting λ. Required by the change-detection step in stage 4.
- Fresh Source Cooperative temp credentials in
.env.json at the repo root. Tokens expire in ~1 hour.
⚠️ Credential refresh check. Source Cooperative uses short-lived AWS
credentials (aws_access_key_id starting with ASIA…). Before running
step 7, ask the user to regenerate them at
https://source.coop/repositories/henryspatialanalysis/openpois/manage
and overwrite .env.json at the repo root. The upload script will warn if
the file looks stale, but it cannot tell whether the token itself has
expired until it actually fails.
Steps
-
Bump versions.conflation and versions.source_coop in config.yaml.
versions.source_coop is the remote folder name — YYYY-MM-DD-vN. Keep
vN at v0; only bump v1, v2, … if you re-upload under the same
calendar date.
-
Review conflation parameters (config.yaml → conflation):
min_match_score (default 0.50) — raises/lowers match acceptancemax_radius_m, default_radius_m — per-label radii come from match_radii.csv- Component weights:
distance_weight, name_weight, type_weight, identifier_weight
- Changing these reshapes match counts — run with
--test first (Seattle bbox).
-
Sync taxonomy if crosswalks changed — run the sync-taxonomy skill. It regenerates site/public/taxonomy.html and site/src/taxonomy.generated.js, and detects drift in the hand-maintained display labels.
-
Run the conflation pipeline. The canonical entry point is make conflate, which orchestrates three sub-steps so every national run gets the OSM-history change-detection penalty automatically (see docs/change-detection.md for the design and tunables):
build_ghosts.py — reconstruct ghost POIs from OSM history (ghosts.parquet under versions.ghost_osm).conflate.py --output-suffix=baseline — OSM × Overture matching, writes conflated_baseline.parquet (no-CD archive).apply_change_detection.py — shadow-match unmatched Overture against the ghosts and apply the per-shared_label δ penalty; writes the canonical conflated.parquet.
make conflate
make conflate TEST=1
make build_ghosts
make conflate_baseline
make apply_cd
Outputs:
conflated.parquet — canonical output that downstream steps consume (CD applied).conflated_baseline.parquet — same shape but without the CD penalty; kept on disk for spot-checks.ghosts.parquet under versions.ghost_osm — see docs/change-detection.md.match_diagnostics.parquet.
-
Match-rate sanity check:
python scripts/conflation/summarize.py
Writes summary_by_label.csv.
-
Partition for web — geohash-4 partition, geohash-6 sort:
python scripts/conflation/format_for_upload.py
Outputs conflated_partitioned/ (and OSM-only osm_snapshot_partitioned/).
6.5. Build PMTiles — single-zoom (z14) archives consumed directly by the
site via ol-pmtiles. Intermediate FlatGeobufs are cleaned up on success.
bash python -u scripts/osm_snapshot/prepare_pmtiles.py \ 2>&1 | tee ~/data/openpois/logs/pmtiles_osm_<version>.log python -u scripts/conflation/prepare_pmtiles.py \ 2>&1 | tee ~/data/openpois/logs/pmtiles_conflated_<version>.log
Properties and zoom range are configured under publish.pmtiles in
config.yaml.
- Publish to Source Cooperative — uploads OSM + conflated parquet,
both PMTiles, and a freshly-rendered per-version
README.md under
<repo>/<versions.source_coop>/. Confirm the credential check above first.
python scripts/publish/upload_to_source_coop.py --dry-run
python -u scripts/publish/upload_to_source_coop.py \
2>&1 | tee ~/data/openpois/logs/publish_<version>.log
python scripts/publish/upload_to_source_coop.py --update-top-level
--skip-osm-parquet, --skip-conflated-parquet, and --skip-pmtiles
allow partial reuploads (e.g. after regenerating PMTiles alone).
Verification
summary_by_label.csv match rates should resemble the prior run; large drifts mean a parameter or crosswalk regression.match_diagnostics.parquet for per-pair forensics on surprising matches.- Spot-check the version landing page at
https://source.coop/henryspatialanalysis/openpois/ and confirm the
per-version
README.md renders with the expected OSM date, Overture
release, and row counts.
- See skills/verify-pipeline-run.
Next
- Always bump the frontend after a successful upload — the site's PMTiles
URLs are pinned to a version folder and do not auto-follow new uploads, so
a refresh isn't complete until the frontend points at the new
versions.source_coop. Treat this as a required step of every monthly refresh,
not an optional follow-up: skills/update-site.
Key code
