hf-datasets

Estrelas12

Forks0

Atualizado21 de abril de 2026 às 21:26

Inspect a HuggingFace dataset in one call — status, configs/splits, schema, sample rows, parquet file listing, and detailed messages-column analysis (roles, keys, tool-calls). REQUIRED before submitting any training job to verify column format matches the training method.

Instalação

Instalar com Codex ou Claude Copie este prompt, cole no Codex, Claude ou outro assistente e deixe que ele revise a página da skill e instale para você.

Fonte

mudler

mudler/universal-ml-intern

Abrir repositório GitHub Ver repositórios do creator

Download

Ocupações relacionadasSOC

Baseado na classificação ocupacional SOC

Cientistas de dadosInformática e Matemática·SOC 15-2051

SKILL.md

readonly

name	hf-datasets
description	Inspect a HuggingFace dataset in one call — status, configs/splits, schema, sample rows, parquet file listing, and detailed messages-column analysis (roles, keys, tool-calls). REQUIRED before submitting any training job to verify column format matches the training method.

hf-datasets

Backed by scripts/inspect_dataset.py.

When to use

Always call this before training. Training fails with KeyError if columns don't match the method. Specifically:

SFT needs messages, text, or prompt + completion.
DPO needs prompt, chosen, rejected.
GRPO needs prompt.

Also use to:

Understand a dataset before writing data-loading code.
Check schema changes between dataset versions.
Analyze the structure of chat-format datasets (which roles are used, whether tool-calls are present).
See an example row before deciding if the dataset fits the task.

Usage

# Full inspection — auto-detects config and split
python scripts/inspect_dataset.py --dataset stanfordnlp/imdb

# Pick a specific split + more sample rows
python scripts/inspect_dataset.py --dataset HuggingFaceH4/ultrachat_200k --split train_sft --sample-rows 5

# Private / gated dataset — HF_TOKEN must be set
HF_TOKEN=hf_... python scripts/inspect_dataset.py --dataset org/gated-dataset

What you get

Output is markdown with sections:

Status — whether viewer / preview / search / filter / statistics are available.
Structure — table of configs × splits.
Schema — table of columns × types (ClassLabel values inlined if ≤5 classes).
Sample Rows — up to --sample-rows rows, long values truncated at 150 chars.
Messages Column Format (auto-included when a column named messages is present) — roles seen, which keys exist (role, content, tool_calls, tool_call_id, name, function_call), tool-call/result presence, and an example message structure.
Files (Parquet) — parquet file count + total size, grouped by config/split.

Flags

--dataset (required) — org/name format.
--config, --split — auto-detected if omitted.
--sample-rows N — default 3, cap 10.
--json — wrap output in {"ok": bool, "output": "..."}.

Environment

HF_TOKEN — only needed for private / gated datasets.

Related skills

hf-papers → find_datasets surfaces datasets linked to a paper. Pipe the top hit into this skill.
hf-repos → for deeper repo-level file access (e.g. read a custom dataset_infos.json).

Mais deste repositório

mesmo repositório

paper-search-alt

mudler/universal-ml-intern

Alternative paper search backends — OpenAlex (~250M works, no auth, strong PDF extraction) and CORE (~49M full-text, BODY search). Use when hf-papers can't find a paper, when Semantic Scholar is rate-limited, when you need PDFs of older/long-tail papers, or when you need to search the actual body text of papers (not just titles/abstracts).

2026-04-2112

github-code

mudler/universal-ml-intern

Discover GitHub repositories, find example scripts within them, and read source files with line-range support and automatic Jupyter notebook conversion. Use BEFORE writing any ML training, fine-tuning, or inference code — your knowledge of library APIs is outdated and example scripts show the current patterns.

2026-04-2112

hf-docs

mudler/universal-ml-intern

Explore and fetch HuggingFace library documentation (transformers, trl, datasets, peft, accelerate, trackio, vllm, optimum, courses, etc.). Use when you need current trainer arguments, config parameters, or API reference that your internal knowledge may have wrong.

2026-04-2112

hf-jobs

mudler/universal-ml-intern

Submit and manage HuggingFace compute jobs — run training / fine-tuning / inference / batch work on HF cloud infrastructure in Python or Docker mode. Supports one-shot and recurring (cron-style) jobs.

2026-04-2112

hf-papers

mudler/universal-ml-intern

Discover ML research papers, read their contents by section, trace citations, find linked datasets/models — combining HuggingFace Papers, arXiv, and Semantic Scholar. Use whenever you need to research a method, find training recipes with published results, crawl a citation graph, or locate the datasets a paper used. This is your starting point for any literature-first workflow.

2026-04-2112

hf-repos

mudler/universal-ml-intern

File and git-like operations on HuggingFace repositories (model / dataset / space). Covers list / read / upload / delete of files, branches, tags, pull requests, and repo management (create / update). Use to inspect repos, ship artifacts, and propose changes via PRs.

2026-04-2112

name	hf-datasets
description	Inspect a HuggingFace dataset in one call — status, configs/splits, schema, sample rows, parquet file listing, and detailed messages-column analysis (roles, keys, tool-calls). REQUIRED before submitting any training job to verify column format matches the training method.

hf-datasets

Backed by scripts/inspect_dataset.py.

When to use

Always call this before training. Training fails with KeyError if columns don't match the method. Specifically:

SFT needs messages, text, or prompt + completion.
DPO needs prompt, chosen, rejected.
GRPO needs prompt.

Also use to:

Understand a dataset before writing data-loading code.
Check schema changes between dataset versions.
Analyze the structure of chat-format datasets (which roles are used, whether tool-calls are present).
See an example row before deciding if the dataset fits the task.

Usage

# Full inspection — auto-detects config and split
python scripts/inspect_dataset.py --dataset stanfordnlp/imdb

# Pick a specific split + more sample rows
python scripts/inspect_dataset.py --dataset HuggingFaceH4/ultrachat_200k --split train_sft --sample-rows 5

# Private / gated dataset — HF_TOKEN must be set
HF_TOKEN=hf_... python scripts/inspect_dataset.py --dataset org/gated-dataset

What you get

Output is markdown with sections:

Status — whether viewer / preview / search / filter / statistics are available.
Structure — table of configs × splits.
Schema — table of columns × types (ClassLabel values inlined if ≤5 classes).
Sample Rows — up to --sample-rows rows, long values truncated at 150 chars.
Messages Column Format (auto-included when a column named messages is present) — roles seen, which keys exist (role, content, tool_calls, tool_call_id, name, function_call), tool-call/result presence, and an example message structure.
Files (Parquet) — parquet file count + total size, grouped by config/split.

Flags

--dataset (required) — org/name format.
--config, --split — auto-detected if omitted.
--sample-rows N — default 3, cap 10.
--json — wrap output in {"ok": bool, "output": "..."}.

Environment

HF_TOKEN — only needed for private / gated datasets.

Related skills

hf-papers → find_datasets surfaces datasets linked to a paper. Pipe the top hit into this skill.
hf-repos → for deeper repo-level file access (e.g. read a custom dataset_infos.json).