dataset-publish

name

dataset-publish

description

Refresh and push the public Hugging Face dataset chayuto/klondike-llm-decisions after the corpus grows. Use this whenever the user wants to "publish the dataset", "update HF", "push to Hugging Face", "refresh the dataset", "re-publish klondike-llm-decisions", or bump the public corpus after a harvest has been ingested. It runs the regenerate, validate, push, verify, record flow end to end: regenerate the publish shards, gate them by id+field, upload only the changed Parquet shards, round-trip verify against the Hub, then append a Push-history row to the canonical log and update the recall memory. The cached HF token is usually invalid (temp tokens get revoked after each push), so the skill expects the user to supply a fresh WRITE token inline and reminds them to revoke it afterwards. This skill is for the PUBLISH step only; bringing new harvester exports into the corpus first is the solitaire-ingest skill's job. ALSO use it for a card-only change (edit the generator, re-push just data/publish/README.md) or to publish/refresh the companion LoRA model repos.

dataset-publish

Push the public dataset chayuto/klondike-llm-decisions to Hugging Face after the local corpus has grown. The full, version-controlled record of what the dataset is, its configs, the storage format, and the push history lives in docs/hf_dataset_publish_log.md. That doc is the source of truth; this skill is the operational runbook that drives it and keeps it current.

Lead with the result the user cares about (new counts, delta vs last push, the Hub commit link). Save the long-form for the log doc and the memory.

Preconditions

• New exports are already ingested. Publishing is a selection over the store, not an ingest. If the user just dropped harvester files, run the solitaire-ingest skill first, then come back here. If nothing new was ingested, scripts/ingest_exports.py is idempotent and writes 0 shards, so a push then uploads nothing, which is the correct no-op.
• A valid HF WRITE token. The cached token at ~/.cache/huggingface/token is usually invalid: these are temporary tokens the user revokes after each push (recorded every time in the memory and the log). Do not assume it works. Check HfApi().whoami() and, if it fails, ask the user for a fresh write token inline. Never persist it to a file or memory; remind the user to revoke it once the push verifies.

The five steps

Source-of-truth detail is in docs/hf_dataset_publish_log.md ("How to refresh"). The flow:

1. Regenerate the publish shards

.venv/bin/python scripts/ingest_exports.py

Idempotent and append-only: it adds only interaction ids not already in a config's tail shard and prints the live row counts per config (full / clean-raw / clean-lean, then the 26b cohort). 0 shard(s) written means the on-disk shards already reflect every ingested file, which is fine (the data may have been regenerated by a prior ingest run); keep going. A new field in the schema needs a one-time --rebuild to re-pack all shards (the script prints a note when it sees a field absent from the frozen schema).

Read off the new counts and compute the delta versus the last Push-history row in the log doc. You will need both for the push note.

2. Gate locally by id and field

.venv/bin/python scripts/validate_shards.py --ref local --src local

Must print INTEGRITY OK for every config. Do not push if it does not.

3. Push only the changed shards

.venv/bin/python -c "
from huggingface_hub import HfApi
info = HfApi(token='<WRITE_TOKEN>').upload_folder(
    folder_path='data/publish',
    repo_id='chayuto/klondike-llm-decisions',
    repo_type='dataset',
    ignore_patterns=['*.jsonl'],
)
print('COMMIT_OID:', info.oid)
print('COMMIT_URL:', info.commit_url)
"

Frozen shards are immutable, so only new/changed tail shards and the card go up (single-digit MB, not the whole corpus). ignore_patterns=['*.jsonl'] keeps any stray JSONL out. The card (data/publish/README.md) is regenerated by render_dataset_card() in scripts/ingest_exports.py; edit the generator, not the published README, and the change flows up on the next push. Capture the printed COMMIT_OID for the push note.

4. Round-trip verify against the Hub

.venv/bin/python scripts/validate_shards.py --ref local --src hub

Must print INTEGRITY OK for every config. This confirms the Hub now matches local by id and by every field, the gate that the push actually landed. It downloads the shards, so it takes a minute; run it in the background if needed.

5. Record it (do not skip)

• Append a row to the Push-history table in docs/hf_dataset_publish_log.md (newest first): date, the Hub commit hash linked to https://huggingface.co/datasets/chayuto/klondike-llm-decisions/commit/<hash>, the per-config counts, and a Notes cell naming the sessions folded in (wins, resigns, the proven-dead / killed loss set, any new model cohort) and the delta versus the prior push. Match the established cell style by reading the last couple of rows first.
• Update the hf-dataset-published memory's "Latest push" line with the new date, commit, counts, and delta, and move the old line to "Prior push". The memory is only a recall pointer; the log doc holds the detail.
• If the card's prose changed (not just auto-emitted counts), note that too.

Style note

The user dislikes em-dashes, unicode minus, emoji, and right-arrow glyphs in anything they read or publish (the feedback_no_ai_slop_style memory). The log doc and card are published-adjacent: write the Push-history note and any card prose with plain hyphens, commas, and "to"/"vs", never an em-dash or an arrow glyph.

Gotchas

• Token is the usual blocker. Expect whoami() to fail; have the user paste a fresh write token. Confirm whoami returns chayuto before pushing.
• 0 shard(s) written is not an error. It means no new ids since the last regenerate. Still validate and (if local differs from the Hub) push; if the Hub already matches, the upload is a clean no-op.
• Never hand-edit data/publish/ or data/SUMMARY.md. Both are generated. Card changes go in render_dataset_card(); stats are auto-emitted.
• The shards and the ledger are gitignored (data/publish/, data/index/publish_shards.json); they are regenerable, so there is nothing to commit for the data itself. What you do commit is the log-doc Push-history row (and any generator/card edits).
• 26b counts can stay flat while full/clean grow. The 26b cohort only moves when a new gemma-4-26b-a4b-it session is ingested. A flat 26b row is expected, not a missed config.

Companion models (same token, different repos)

The two LoRA advisor adapters live in separate model repos and are pushed the same way (HfApi(token=...).upload_folder(..., repo_type='model')). The how-to, the per-repo metadata conventions, and the model Push-history table are all in docs/hf_dataset_publish_log.md under "Companion models". For a card-only change, re-upload README.md alone with api.upload_file(...). Staging dirs (gemma4_finetune/publish_hf*/) are gitignored.

Related

• Canonical record + how-to + history: docs/hf_dataset_publish_log.md
• Bringing new exports into the corpus first: the solitaire-ingest skill
• Generator / card / shard writer: scripts/ingest_exports.py (render_dataset_card(), publish_sharded())
• Integrity gate: scripts/validate_shards.py
• Upload-efficiency rationale (why sharded, append-only): docs/reports/20260606_hf_upload_efficiency_research.md