| name | dataset-publish |
| description | Refresh and push the public Hugging Face dataset chayuto/klondike-llm-decisions after the corpus grows. Use this whenever the user wants to "publish the dataset", "update HF", "push to Hugging Face", "refresh the dataset", "re-publish klondike-llm-decisions", or bump the public corpus after a harvest has been ingested. It runs the regenerate, validate, push, verify, record flow end to end: regenerate the publish shards, gate them by id+field, upload only the changed Parquet shards, round-trip verify against the Hub, then append a Push-history row to the canonical log and update the recall memory. The cached HF token is usually invalid (temp tokens get revoked after each push), so the skill expects the user to supply a fresh WRITE token inline and reminds them to revoke it afterwards. This skill is for the PUBLISH step only; bringing new harvester exports into the corpus first is the solitaire-ingest skill's job. ALSO use it for a card-only change (edit the generator, re-push just data/publish/README.md) or to publish/refresh the companion LoRA model repos. |
dataset-publish
Push the public dataset chayuto/klondike-llm-decisions to Hugging Face after
the local corpus has grown. The full, version-controlled record of what the
dataset is, its configs, the storage format, and the push history lives in
docs/hf_dataset_publish_log.md. That doc is the source of truth; this skill is
the operational runbook that drives it and keeps it current.
Lead with the result the user cares about (new counts, delta vs last push, the
Hub commit link). Save the long-form for the log doc and the memory.
Preconditions
- New exports are already ingested. Publishing is a selection over the
store, not an ingest. If the user just dropped harvester files, run the
solitaire-ingest skill first, then come back here. If nothing new was
ingested,
scripts/ingest_exports.py is idempotent and writes 0 shards, so a
push then uploads nothing, which is the correct no-op.
- A valid HF WRITE token. The cached token at
~/.cache/huggingface/token
is usually invalid: these are temporary tokens the user revokes after each
push (recorded every time in the memory and the log). Do not assume it works.
Check HfApi().whoami() and, if it fails, ask the user for a fresh write
token inline. Never persist it to a file or memory; remind the user to revoke
it once the push verifies.
The five steps
Source-of-truth detail is in docs/hf_dataset_publish_log.md ("How to
refresh"). The flow:
1. Regenerate the publish shards
.venv/bin/python scripts/ingest_exports.py
Idempotent and append-only: it adds only interaction ids not already in a
config's tail shard and prints the live row counts per config
(full / clean-raw / clean-lean, then the 26b cohort). 0 shard(s) written
means the on-disk shards already reflect every ingested file, which is fine (the
data may have been regenerated by a prior ingest run); keep going. A new
field in the schema needs a one-time --rebuild to re-pack all shards (the
script prints a note when it sees a field absent from the frozen schema).
Read off the new counts and compute the delta versus the last Push-history row
in the log doc. You will need both for the push note.
2. Gate locally by id and field
.venv/bin/python scripts/validate_shards.py --ref local --src local
Must print INTEGRITY OK for every config. Do not push if it does not.
3. Push only the changed shards
.venv/bin/python -c "
from huggingface_hub import HfApi
info = HfApi(token='<WRITE_TOKEN>').upload_folder(
folder_path='data/publish',
repo_id='chayuto/klondike-llm-decisions',
repo_type='dataset',
ignore_patterns=['*.jsonl'],
)
print('COMMIT_OID:', info.oid)
print('COMMIT_URL:', info.commit_url)
"
Frozen shards are immutable, so only new/changed tail shards and the card go up
(single-digit MB, not the whole corpus). ignore_patterns=['*.jsonl'] keeps any
stray JSONL out. The card (data/publish/README.md) is regenerated by
render_dataset_card() in scripts/ingest_exports.py; edit the generator, not
the published README, and the change flows up on the next push. Capture the
printed COMMIT_OID for the push note.
4. Round-trip verify against the Hub
.venv/bin/python scripts/validate_shards.py --ref local --src hub
Must print INTEGRITY OK for every config. This confirms the Hub now matches
local by id and by every field, the gate that the push actually landed. It
downloads the shards, so it takes a minute; run it in the background if needed.
5. Record it (do not skip)
- Append a row to the Push-history table in
docs/hf_dataset_publish_log.md
(newest first): date, the Hub commit hash linked to
https://huggingface.co/datasets/chayuto/klondike-llm-decisions/commit/<hash>,
the per-config counts, and a Notes cell naming the sessions folded in (wins,
resigns, the proven-dead / killed loss set, any new model cohort) and the
delta versus the prior push. Match the established cell style by reading the
last couple of rows first.
- Update the
hf-dataset-published memory's "Latest push" line with the new
date, commit, counts, and delta, and move the old line to "Prior push". The
memory is only a recall pointer; the log doc holds the detail.
- If the card's prose changed (not just auto-emitted counts), note that too.
Style note
The user dislikes em-dashes, unicode minus, emoji, and right-arrow glyphs in
anything they read or publish (the feedback_no_ai_slop_style memory). The log
doc and card are published-adjacent: write the Push-history note and any card
prose with plain hyphens, commas, and "to"/"vs", never an em-dash or an arrow
glyph.
Gotchas
- Token is the usual blocker. Expect
whoami() to fail; have the user paste
a fresh write token. Confirm whoami returns chayuto before pushing.
0 shard(s) written is not an error. It means no new ids since the last
regenerate. Still validate and (if local differs from the Hub) push; if the
Hub already matches, the upload is a clean no-op.- Never hand-edit
data/publish/ or data/SUMMARY.md. Both are generated.
Card changes go in render_dataset_card(); stats are auto-emitted.
- The shards and the ledger are gitignored (
data/publish/,
data/index/publish_shards.json); they are regenerable, so there is nothing
to commit for the data itself. What you do commit is the log-doc
Push-history row (and any generator/card edits).
- 26b counts can stay flat while full/clean grow. The 26b cohort only moves
when a new
gemma-4-26b-a4b-it session is ingested. A flat 26b row is
expected, not a missed config.
Companion models (same token, different repos)
The two LoRA advisor adapters live in separate model repos and are pushed the
same way (HfApi(token=...).upload_folder(..., repo_type='model')). The
how-to, the per-repo metadata conventions, and the model Push-history table are
all in docs/hf_dataset_publish_log.md under "Companion models". For a
card-only change, re-upload README.md alone with api.upload_file(...).
Staging dirs (gemma4_finetune/publish_hf*/) are gitignored.
Related
- Canonical record + how-to + history:
docs/hf_dataset_publish_log.md
- Bringing new exports into the corpus first: the solitaire-ingest skill
- Generator / card / shard writer:
scripts/ingest_exports.py
(render_dataset_card(), publish_sharded())
- Integrity gate:
scripts/validate_shards.py
- Upload-efficiency rationale (why sharded, append-only):
docs/reports/20260606_hf_upload_efficiency_research.md
