hf-ds

name

hf-ds

description

Work with the Hugging Face `datasets` Python library the right way — load and stream datasets, inspect schema/size without downloading, cast Features, and build or publish datasets to the Hub. Use when loading a dataset (load_dataset), streaming a large one (IterableDataset), previewing rows, or pushing local files/folders to the Hub (push_to_hub). Pairs with the `hf ds` CLI extension for safe one-shot operations.

Hugging Face datasets (Python) + the hf ds CLI

Use the datasets library without the common footguns. For deterministic one-shot operations, prefer the tested hf ds CLI extension; for anything bespoke, write Python (patterns below).

Install the CLI extension: hf extensions install davanstrien/hf-ds → then hf ds ....

CLI (safe one-shot ops — tested, drift-guarded)

Three verbs: inspect → preview → publish.

• hf ds inspect <repo> — configs / splits / schema / size, no download
• hf ds head <repo> [-n 5] — preview rows by streaming, no download. Media cells (images, audio, arrays) show as compact descriptors like {"_type":"image","mode":"L","size":[28,28]}, not raw bytes; --json for faithful nested JSON, --format/--no-truncate for table control.
• hf ds push <local> <repo> — publish local files/folders as a proper Dataset (auto-detects the loader, embeds media). --dry-run pre-flights the source against Hub limits (per-file size, 10k files/folder, empty) before the build and previews without uploading. On a bespoke layout no loader fits, push prints the canonical from_generator → push_to_hub recipe to adapt.

Loading & streaming (stable patterns)

• load_dataset("repo") downloads and caches the whole dataset. For large data, stream: ds = load_dataset("repo", split="train", streaming=True) → an IterableDataset.
• An IterableDataset has no random access. First row: next(iter(ds)) or ds.take(1) — not ds[0] (in current datasets that returns a lazy column accessor, not a row) and not len(ds) (raises). hf ds head does this correctly for you.
• Inspect before downloading: get_dataset_config_names(repo), get_dataset_split_names(repo, cfg), get_dataset_config_info(repo, cfg) (features + size, no data). Or just hf ds inspect <repo>.
• trust_remote_code is gone — modern Hub datasets are standard formats only.

Publishing local data

• Images/audio in class subfolders or with a metadata.csv: hf ds push ./dir you/repo (class subfolders → ClassLabel; media bytes embed on push so the Viewer renders).

• Tabular/text files: hf ds push ./data.csv you/repo (csv / json / parquet / text / ...).

• Bespoke format no loader fits — the generator escape hatch (hf ds push prints this on failure):

  from datasets import Dataset  # add Features(...) / Image() / Audio() as needed
  def gen():
      for item in your_source():
          yield {"text": item["text"], "label": item["label"]}
  Dataset.from_generator(gen).push_to_hub("you/repo")
  

• Always finish with push_to_hub (writes parquet + configures the Viewer).

Related skills

• Zero-Python / REST exploration (paginate rows, search, filter): the huggingface-datasets skill.
• General hf CLI usage: the hf-cli skill.