| name | hf-datasets |
| description | Explore, fetch, and create Hugging Face datasets. Use the Dataset Viewer API (splits, rows, search, filter, parquet, size, statistics) to inspect data without downloading, and create or upload datasets to the Hub. Use for building or inspecting training data.
|
| stack | ml |
Hugging Face Datasets
Two surfaces: the Dataset Viewer API (read-only inspection, no download)
and the Hub (create/upload). The hf datasets CLI and datasets Python
library sit on top.
Dataset Viewer API (read-only)
Base URL: https://datasets-server.huggingface.co. GET, URL-encoded params.
Gated/private datasets need Authorization: Bearer <HF_TOKEN>.
| Purpose | Endpoint |
|---|
| Validate availability | /is-valid?dataset=<ns/repo> |
| List configs + splits | /splits?dataset=<ns/repo> |
| Preview rows | /first-rows?dataset=...&config=...&split=... |
| Paginate rows (max 100) | /rows?dataset=...&config=...&split=...&offset=0&length=100 |
| Full-text search | /search?dataset=...&split=...&query=<text> |
| Row predicates | /filter?dataset=...&split=...&where=<pred>&orderby=<col> |
| Parquet shard URLs | /parquet?dataset=<ns/repo> |
| Size totals | /size?dataset=<ns/repo> |
| Column statistics | /statistics?dataset=...&config=...&split=... |
Pagination: read num_rows_total / num_rows_per_page / partial from the
response to drive the next offset.
curl "https://datasets-server.huggingface.co/rows?dataset=stanfordnlp/imdb&config=plain_text&split=train&offset=0&length=100"
CLI (faster for some tasks)
hf datasets info <id>
hf datasets parquet <id>
hf datasets sql "<duckdb>"
Create / upload a dataset
- Structure data as parquet/CSV/jsonl, or use the
datasets library to build
an Arrow dataset.
hf upload <ns>/<dataset-name> <data-dir> --type dataset --commit-message "...".- Write a dataset card (
README.md) with task, columns, license, and a
datasets tag so it's discoverable.
When to use
Inspecting a dataset without downloading it, or creating/uploading a dataset
to the Hub (e.g. NANOG1 multimodal oncology data).
Pitfalls
- The Viewer API is read-only; it never downloads. Use
hf download for the
actual files.
- For data quality judgment (missing/duplicate/sensitive values), pair with
data/data-hygiene.
Adapted from huggingface/skills (Apache-2.0).
