메뉴
Submit and manage HuggingFace compute jobs — run training / fine-tuning / inference / batch work on HF cloud infrastructure in Python or Docker mode. Supports one-shot and recurring (cron-style) jobs.
Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.
SOC 직업 분류 기준
Alternative paper search backends — OpenAlex (~250M works, no auth, strong PDF extraction) and CORE (~49M full-text, BODY search). Use when hf-papers can't find a paper, when Semantic Scholar is rate-limited, when you need PDFs of older/long-tail papers, or when you need to search the actual body text of papers (not just titles/abstracts).
Discover GitHub repositories, find example scripts within them, and read source files with line-range support and automatic Jupyter notebook conversion. Use BEFORE writing any ML training, fine-tuning, or inference code — your knowledge of library APIs is outdated and example scripts show the current patterns.
Inspect a HuggingFace dataset in one call — status, configs/splits, schema, sample rows, parquet file listing, and detailed messages-column analysis (roles, keys, tool-calls). REQUIRED before submitting any training job to verify column format matches the training method.
Explore and fetch HuggingFace library documentation (transformers, trl, datasets, peft, accelerate, trackio, vllm, optimum, courses, etc.). Use when you need current trainer arguments, config parameters, or API reference that your internal knowledge may have wrong.
Discover ML research papers, read their contents by section, trace citations, find linked datasets/models — combining HuggingFace Papers, arXiv, and Semantic Scholar. Use whenever you need to research a method, find training recipes with published results, crawl a citation graph, or locate the datasets a paper used. This is your starting point for any literature-first workflow.
File and git-like operations on HuggingFace repositories (model / dataset / space). Covers list / read / upload / delete of files, branches, tags, pull requests, and repo management (create / update). Use to inspect repos, ship artifacts, and propose changes via PRs.
| name | hf-jobs |
| description | Submit and manage HuggingFace compute jobs — run training / fine-tuning / inference / batch work on HF cloud infrastructure in Python or Docker mode. Supports one-shot and recurring (cron-style) jobs. |
Backed by scripts/hf_jobs.py. Wraps huggingface_hub.HfApi job endpoints.
Before submitting any non-trivial job, confirm:
github-code to find it.)hf-datasets to confirm.)hf-repos to list files.)push_to_hub=True AND hub_model_id= set. Job storage is ephemeral; without these, the trained model is gone.For batch / ablation runs: submit one job first, confirm it starts training correctly, then submit the rest. Do not submit them all at once — if there's a bug, they all fail for the same reason.
| Subcommand | What it does |
|---|---|
run | Run a one-shot job (Python script or Docker command) |
ps | List jobs (default: running only; --all for everything) |
logs | Fetch full logs for a job (after it completes) |
inspect | JSON details for a job |
cancel | Cancel a running job |
sched-run | Create a scheduled (cron) job |
sched-ps | List scheduled jobs |
sched-inspect | Inspect a scheduled job |
sched-delete | Delete a scheduled job |
sched-suspend | Suspend (pause) a scheduled job |
sched-resume | Resume a suspended scheduled job |
Python training job (recommended: pass a local file or a URL, not inline code):
python scripts/hf_jobs.py run \
--script ./train.py \
--dep transformers --dep trl --dep torch --dep datasets --dep trackio --dep accelerate \
--hardware-flavor a100-large \
--timeout 8h \
--env WANDB_DISABLED=true
Docker one-shot:
python scripts/hf_jobs.py run \
--command duckdb -c 'select 1 + 2' \
--image duckdb/duckdb \
--hardware-flavor cpu-basic \
--timeout 1h
Inline Python via stdin (rare — prefer a file):
cat << 'EOF' | python scripts/hf_jobs.py run --script-stdin --dep trl --hardware-flavor a10g-large --timeout 4h
# your script here
EOF
Submit and return immediately (don't block on log stream):
python scripts/hf_jobs.py run --script ./train.py --hardware-flavor a100-large --timeout 8h --no-wait
# → returns job ID
python scripts/hf_jobs.py ps
python scripts/hf_jobs.py logs --job-id <id>
Scheduled:
python scripts/hf_jobs.py sched-run \
--script ./nightly_eval.py \
--schedule '@daily' \
--hardware-flavor t4-small \
--timeout 2h
| Flavor | vCPU / RAM / GPU |
|---|---|
cpu-basic | 2 / 16 GB |
cpu-upgrade | 8 / 32 GB |
t4-small | 4 / 15 GB / 16 GB |
a10g-small | 4 / 15 GB / 24 GB |
a10g-large | 12 / 46 GB / 24 GB |
a10g-largex2 | 24 / 92 GB / 48 GB |
a100-large | 12 / 142 GB / 80 GB |
a100x4 | 48 / 568 GB / 320 GB |
a100x8 | 96 / 1136 GB / 640 GB |
l40sx4 | 48 / 382 GB / 192 GB |
Sizing: 1–3B → a10g-largex2, 7–13B → a100-large, 30B+ → l40sx4 / a100x4, 70B+ → a100x8.
a10g-small and a10g-large share the same 24 GB GPU — only CPU/RAM differ.
per_device_train_batch_size, raise gradient_accumulation_steps proportionally (effective batch stays the same).gradient_checkpointing=True.Do not switch SFT → LoRA or reduce max_length. Those change what the user asked for.
HF_TOKEN — required. Auto-injected into the job's HF_TOKEN + HUGGINGFACE_HUB_TOKEN secrets.HF_NAMESPACE — optional. Defaults to whoami().name.github-code — to find the reference implementation before writing the training script.hf-datasets — to verify dataset format before submission.hf-repos — to confirm the base model exists and has the right files.