hf-dataset-publishing

name

hf-dataset-publishing

description

Save/publish analysis or computation results from ANY ecosystem repo to Hugging Face as a queryable, viewer-renderable dataset. Use when the user wants to "save results to hugging face", "publish dataset to HF", "hugging face data saving", "save analysis results", "hf dataset", "make results queryable", or "render via datasets-server API". Reshapes nested results into flat parquet tables, writes a dataset card with a viewer `configs:` block and provenance, applies license/public-vs-private routing, enforces a domain data-quality gate (faithful-to-source != correct), publishes to `aceengineer/<repo>-<projection>`, and verifies via the datasets-server API.

Publishing Analysis Results to Hugging Face

Turn the results of any analysis/computation (from any repo in this ecosystem) into a queryable, auto-rendered Hugging Face dataset. HF's datasets-server auto-indexes parquet tables and exposes them both in the in-browser viewer and via a REST API — so a published dataset becomes a live, linkable, machine-queryable projection of your results.

This skill encodes a workflow that was executed end-to-end on 2026-07-11. Follow THIS, not generic HF docs.

When to use

• You have analysis output (JSON, dicts, dataframes, a results folder) and want to save it somewhere queryable and viz-renderable, not just a file on disk.
• You want a shareable, indexable surface for a set of computed results (per field / per well / per country / per run-summary, etc.).
• Someone says "publish this to Hugging Face" / "save these results as an HF dataset".

Do NOT use this to record algorithm-run identity/replay ledgers — that is the separate aceengineer/<repo>-runs contract (workspace-hub#3433). See Conventions below.

Prerequisites (environment-backed auth)

• CLI is hf (huggingface-cli is DEPRECATED — do not use it).
• Token lives at ~/.cache/huggingface/token. Verify:

  hf auth whoami          # expect: user=<u> and orgs include aceengineer
  

• Python libs available: huggingface_hub (1.21+), pandas, pyarrow.
• Never print, echo, log, or commit the token. Secrets stay in env / token-file only.

The 4-step workflow

Step 1 — Reshape nested results → flat tabular tables (one per entity)

Load the analysis output and build one pandas.DataFrame per entity type (e.g. fields, wells, countries). For each table select SCALAR columns only:

• Flatten a few useful nested scalars (e.g. econ.breakeven -> econ_breakeven).
• SKIP list/dict-valued columns (they don't render in the tabular viewer).
• Sanitize NaN/inf. The source JSON may carry literal NaN/Infinity tokens that break strict JSON consumers. Parquet stores genuine nulls natively, so coerce non-finite floats to null and write parquet — which is exactly what the HF datasets-server auto-renders.

import math, pandas as pd, numpy as np

def clean(df: pd.DataFrame) -> pd.DataFrame:
    # non-finite floats -> genuine null (parquet-native)
    return df.replace([np.inf, -np.inf], np.nan)

df_fields = clean(pd.DataFrame(field_rows))         # one row per field
df_fields.to_parquet("out/fields.parquet", index=False)

CSV is an acceptable fallback if parquet is unavailable, but parquet is preferred (native nulls

• auto-render).

Step 2 — Dataset card README.md with YAML frontmatter

The frontmatter MUST include license: (see routing rule), pretty_name, tags, and a configs: block listing each parquet table as its own viewer config:

---
license: cc-by-4.0
pretty_name: World Energy Data — Explorer Projection
tags:
  - energy
  - oil-and-gas
  - analysis-results
configs:
  - config_name: fields
    data_files: fields.parquet
  - config_name: wells
    data_files: wells.parquet
  - config_name: countries
    data_files: countries.parquet
---

The card body must cover:

• What it is — one-paragraph description of the projection and its source analysis.
• A table of configs + row counts (e.g. fields — 56 rows).
• Provenance — source file sha256 hashes + a schema_version. This is how a consumer proves which analysis run produced the data.
• Data basis — what the numbers mean, units, and the note that nulls are genuine (not missing-by-accident) because non-finite values were coerced to null.
• Any withheld columns (see the data-quality gate) with a link to the filed issue.

Step 3 — Publish

from huggingface_hub import HfApi
api = HfApi()
repo_id = "aceengineer/<repo>-<projection>"          # e.g. aceengineer/worldenergydata-explorer
api.create_repo(repo_id=repo_id, repo_type="dataset",
                private=<per routing>, exist_ok=True)
api.upload_folder(folder_path="out", repo_id=repo_id, repo_type="dataset",
                  commit_message="Publish <projection> analysis projection")

The bundled publish_analysis_to_hf.py does exactly this (plus auth check + verify) — prefer it over hand-rolling.

Step 4 — Verify (the render-via-API surface)

api.list_repo_files(repo_id=repo_id, repo_type="dataset")   # confirm parquet + README landed

Then hit the datasets-server API (the surface that powers the in-browser viewer):

https://datasets-server.huggingface.co/is-valid?dataset=<repo_id>
https://datasets-server.huggingface.co/rows?dataset=<repo_id>&config=<table>&split=train&offset=0&length=10

Indexing lag is normal. A brand-new dataset returns "server is busier than usual / retry later" for a few minutes before /rows and the in-browser viewer come up. Poll — do not assume failure. Once /is-valid reports valid and /rows returns data, the viewer is live.

Conventions the skill MUST encode

• Naming: aceengineer/<repo>-<projection> — e.g. worldenergydata-explorer. Datasets on the aceengineer org are free.
• Keep SEPARATE from the run-dataset contract. aceengineer/<repo>-runs is workspace-hub#3433's contract-managed algorithm-run ledger (run-identity + replay). Analysis-result projections are a different artifact and get their own dataset. Do not dump projections into -runs.
• License / public-vs-private routing (cite .claude/rules/codes-standards-data-routing.md):
  • Public-domain federal data (BSEE / NOAA / USGS) → public, license: cc-by-4.0.
  • Vendor-licensed / private / client data → must NOT be published to a public HF dataset. A private repo is OK only with explicit owner sign-off.
  • Synthetic / own-analysis output → publisher's choice. Calculations derived from codes and standards are own analysis, not republished standard text — publishable.
  • If unsure of the source's provenance, default to private and ask the owner.
  • Visibility is enforced on every publish, not just at creation (workspace-hub#3483). Re-publishing with --public now flips an existing private dataset, and the tool reads the visibility back off the remote and exits non-zero if it disagrees with what you asked for. The manual update_repo_settings(...) workaround is no longer needed.

DATA-QUALITY GATE (load-bearing — do not skip)

Faithful to source != correct.

Row counts matching the source proves nothing about whether the numbers are right. Before publishing, run a domain plausibility check on the actual values — publishing to a new, labeled, queryable surface is a natural audit trigger, so audit here.

• If values are implausible, withhold those columns, file a cat:data / bug issue, and note the withholding in the card (with the issue link).
• Real case (2026-07-11): economics columns showed $380 breakevens on producing fields — clearly wrong. Those columns were withheld pending worldenergydata#971 rather than published as fact.
• Honesty principle: missing/withheld data → visible provenance/placeholder, never silent omission.

Sandbox gotchas (each cost real cycles)

• python3 -c "..." is DENIED → use a heredoc: python3 - <<'EOF' ... EOF.
• base64 -d is DENIED → fetch repo file content raw instead of decoding the contents API:

  gh api -H "Accept: application/vnd.github.raw" \
    "repos/<owner>/<repo>/contents/<path>?ref=main"
  

• Post-publish, raw.githubusercontent.com and the HF CDN can lag → verify via gh api / HfApi / the datasets-server API, not the raw CDN.
• Org Gradio/Docker Spaces need a paid HF plan (datasets are free). Relevant only if this data later gets a viz Space — the dataset itself costs nothing.

Rebuild-on-publish (C5, workspace-hub#3488)

Publishing a registered dataset (one that has a config/capabilities.yaml entry in aceengineer-website) should refresh the live site automatically — "live" data without a live runtime dependency.

• save_results_to_hf.py POSTs a Vercel Deploy Hook after a verified publish, so the site rebuilds and picks up the new rows (the build re-fetches HF and re-bakes snapshots).
• The hook URL is a secret, read from $VERCEL_DEPLOY_HOOK_URL — never committed. Setting that env var on a publishing host is the opt-in ("publishes from here refresh the site"). Unset → the publish still succeeds, it just doesn't trigger a rebuild.
• Suppress per-run with --no-deploy-hook. A failed/absent hook never fails the publish.
• Backstop: aceengineer-website/.github/workflows/nightly-rebuild.yml triggers a nightly rebuild (covers datasets updated out-of-band), reading the same URL from a GH Actions secret.
• Owner setup (once): Vercel → project Settings → Git → Deploy Hooks → create a main hook → store the URL as $VERCEL_DEPLOY_HOOK_URL (publishing hosts) and as the VERCEL_DEPLOY_HOOK_URL GitHub Actions secret in aceengineer-website.

Bundled resources

• publish_analysis_to_hf.py — reusable helper: verifies hf auth whoami, creates the dataset repo, uploads a prepared folder (parquet + README), then polls the datasets-server /is-valid to report readiness. Parameterized --repo-id / --folder / --private, with a --dry-run. Run python3 publish_analysis_to_hf.py --help.
• PROMPT.md — a copy-paste, parameterized prompt to hand any agent so it runs this whole flow (reshape → card+provenance → sanity-check → publish → verify) while honoring license routing, the withhold-and-file-issue data-quality gate, and the keep-separate-from--runs convention.