hugging-face-vision-trainer

# Vision Model Training on Hugging Face Jobs Train object detection, image classification, and SAM/SAM2 segmentation models on managed cloud GPUs. No local GPU setup required—results are automatically saved to the Hugging Face Hub. ## When to Use This Skill Use this skill when users want to: - Fine-tune object detection models (D-FINE, RT-DETR v2, DETR, YOLOS) on cloud GPUs or local - Fine-tune image classification models (timm: MobileNetV3, MobileViT, ResNet, ViT/DINOv3, or any Transformers classifier) on cloud GPUs or local - Fine-tune SAM or SAM2 models for segmentation / image matting using bbox or point prompts - Train bounding-box detectors on custom datasets - Train image classifiers on custom datasets - Train segmentation models on custom mask datasets with prompts - Run vision training jobs on Hugging Face Jobs infrastructure - Ensure trained vision models are permanently saved to the Hub ## Related Skills - **`hugging-face-jobs`** — General HF Jobs infrastructure: token authentication, hardware flavors, timeout management, cost estimation, secrets, environment variables, scheduled jobs, and result persistence. **Refer to the Jobs skill for any non-training-specific Jobs questions** (e.g., "how do secrets work?", "what hardware is available?", "how do I pass tokens?"). - **`hugging-face-model-trainer`** — TRL-based language model training (SFT, DPO, GRPO). Use that skill for text/language model fine-tuning. ## Local Script Execution Helper scripts use PEP 723 inline dependencies. Run them with `uv run`: ```bash uv run scripts/dataset_inspector.py --dataset username/dataset-name --split train uv run scripts/estimate_cost.py --help ``` ## Prerequisites Checklist Before starting any training job, verify: ### Account & Authentication - Hugging Face Account with [Pro](https://hf.co/pro), [Team](https://hf.co/enterprise), or [Enterprise](https://hf.co/enterprise) plan (Jobs require paid plan) - Authenticated login: Check with `hf_whoami()` (tool) or `hf auth whoami` (terminal) - Token has **write** permissions - **MUST pass token in job secrets** — see directive #3 below for syntax (MCP tool vs Python API) ### Dataset Requirements — Object Detection - Dataset must exist on Hub - Annotations must use the `objects` column with `bbox`, `category` (and optionally `area`) sub-fields - Bboxes can be in **xywh (COCO)** or **xyxy (Pascal VOC)** format — auto-detected and converted - Categories can be **integers or strings** — strings are auto-remapped to integer IDs - `image_id` column is **optional** — generated automatically if missing - **ALWAYS validate unknown datasets** before GPU training (see Dataset Validation section) ### Dataset Requirements — Image Classification - Dataset must exist on Hub - Must have an **`image` column** (PIL images) and a **`label` column** (integer class IDs or strings) - The label column can be `ClassLabel` type (with names) or plain integers/strings — strings are auto-remapped - Common column names auto-detected: `label`, `labels`, `class`, `fine_label` - **ALWAYS validate unknown datasets** before GPU training (see Dataset Validation section) ### Dataset Requirements — SAM/SAM2 Segmentation - Dataset must exist on Hub - Must have an **`image` column** (PIL images) and a **`mask` column** (binary ground-truth segmentation mask) - Must have a **prompt** — either: - A **`prompt` column** with JSON containing `{"bbox": [x0,y0,x1,y1]}` or `{"point": [x,y]}` - OR a dedicated **`bbox`** column with `[x0,y0,x1,y1]` values - OR a dedicated **`point`** column with `[x,y]` or `[[x,y],...]` values - Bboxes should be in **xyxy** format (absolute pixel coordinates) - Example dataset: `merve/MicroMat-mini` (image matting with bbox prompts) - **ALWAYS validate unknown datasets** before GPU training (see Dataset Validation section) ### Critical Settings - **Timeout must exceed expected training time** — Default 30min is TOO SHORT. See directive #6 for recommended values. - **Hub push must be enabled** — `push_to_hub=True`, `hub_model_id="username/model-name"`, token in `secrets` ## Dataset Validation **Validate dataset format BEFORE launching GPU training to prevent the #1 cause of training failures: format mismatches.** **ALWAYS validate for** unknown/custom datasets or any dataset you haven't trained with before. **Skip for** `cppe-5` (the default in the training script). ### Running the Inspector **Option 1: Via HF Jobs (recommended — avoids local SSL/dependency issues):** ```python hf_jobs("uv", { "script": "path/to/dataset_inspector.py", "script_args": ["--dataset", "username/dataset-name", "--split", "train"] }) ``` **Option 2: Locally:** ```bash uv run scripts/dataset_inspector.py --dataset username/dataset-name --split train ``` **Option 3: Via `HfApi().run_uv_job()` (if hf_jobs MCP unavailable):** ```python from huggingface_hub import HfApi api = HfApi() api.run_uv_job( script="scripts/dataset_inspector.py", script_args=["--dataset", "username/dataset-name", "--split", "train"], flavor="cpu-basic", timeout=300, ) ``` ### Reading Results - **`✓ READY`** — Dataset is compatible, use directly - **`✗ NEEDS FORMATTING`** — Needs preprocessing (mapping code provided in output) ## Automatic Bbox Preprocessing The object detection training script (`scripts/object_detection_training.py`) automatically handles bbox format detection (xyxy→xywh conversion), bbox sanitization, `image_id` generation, string category→integer remapping, and dataset truncation. **No manual preprocessing needed** — just ensure the dataset has `objects.bbox` and `objects.category` columns. ## Training workflow Copy this checklist and track progress: ``` Training Progress: - [ ] Step 1: Verify prerequisites (account, token, dataset) - [ ] Step 2: Validate dataset format (run dataset_inspector.py) - [ ] Step 3: Ask user about dataset size and validation split - [ ] Step 4: Prepare training script (OD: scripts/object_detection_training.py, IC: scripts/image_classification_training.py, SAM: scripts/sam_segmentation_training.py) - [ ] Step 5: Save script locally, submit job, and report details ``` **Step 1: Verify prerequisites** Follow the Prerequisites Checklist above. **Step 2: Validate dataset** Run the dataset inspector BEFORE spending GPU time. See "Dataset Validation" section above. **Step 3: Ask user preferences** ALWAYS use the AskUserQuestion tool with option-style format: ```python AskUserQuestion({ "questions": [ { "question": "Do you want to run a quick test with a subset of the data first?", "header": "Dataset Size", "options": [ {"label": "Quick test run (10% of data)", "description": "Faster, cheaper (~30-60 min, ~$2-5) to validate setup"}, {"label": "Full dataset (Recommended)", "description": "Complete training for best model quality"} ], "multiSelect": false }, { "question": "Do you want to create a validation split from the training data?", "header": "Split data", "options": [ {"label": "Yes (Recommended)", "description": "Automatically split 15% of training data for validation"}, {"label": "No", "description": "Use existing validation split from dataset"} ], "multiSelect": false }, { "question": "Which GPU hardware do you want to use?", "header": "Hardware Flavor", "options": [ {"label": "t4-small ($0.40/hr)", "description": "1x T4, 16 GB VRAM — sufficient for all OD models under 100M params"}, {"label": "l4x1 ($0.80/hr)", "description": "1x L4, 24 GB VRAM — more headroom for large images or batch sizes"}, {"label": "a10g-large ($1.50/hr)", "description": "1x A10G, 24 GB VRAM — faster training, more CPU/RAM"}, {"label": "a100-large ($2.50/hr)", "description": "1x A100, 80 GB VRAM — fastest, for very large datasets or image sizes"} ], "multiSelect": false } ] }) ``` **Step 4: Prepare training script** For object detection, use [scripts/object_detection_training.py](scripts/object_detection_training.py) as the production-ready template. For image classification, use [scripts/image_classification_training.py](scripts/image_classification_training.py). For SAM/SAM2 segmentation, use [scripts/sam_segmentation_training.py](scripts/sam_segmentation_training.py). All scripts use `HfArgumentParser` — all configuration is passed via CLI arguments in `script_args`, NOT by editing Python variables. For timm model details, see [references/timm_trainer.md](references/timm_trainer.md). For SAM2 training details, see [references/finetune_sam2_trainer.md](references/finetune_sam2_trainer.md). **Step 5: Save script, submit job, and report** 1. **Save the script locally** to `submitted_jobs/` in the workspace root (create if needed) with a descriptive name like `training_<dataset>_<YYYYMMDD_HHMMSS>.py`. Tell the user the path. 2. **Submit** using `hf_jobs` MCP tool (preferred) or `HfApi().run_uv_job()` — see directive #1 for both methods. Pass all config via `script_args`. 3. **Report** the job ID (from `.id` attribute), monitoring URL, Trackio dashboard (`https://huggingface.co/spaces/{username}/trackio`), expected time, and estimated cost. 4. **Wait for user** to request status checks — don't poll automatically. Training jobs run asynchronously and can take hours. ## Critical directives These rules prevent common failures. Follow them exactly. ### 1. Job submission: `hf_jobs` MCP tool vs Python API **`hf_jobs()` is an MCP tool, NOT a Python function.** Do NOT try to import it from `huggingface_hub`. Call it as a tool: ``` hf_jobs("uv", {"script": training_script_content, "flavor": "a10g-large", "timeout": "4h", "secrets": {"HF_TOKEN": "$HF_TOKEN"}}) ``` **If `hf_jobs` MCP tool is unavailable**, use the Python API directly: ```python from huggingface_hub import HfApi, get_token api = HfApi() job_info = api.run_uv_job( script="path/to/training_script.py", # file PATH, NOT content script_args=["--dataset_name", "cppe-5", ...], flavor="a10g-large", timeout=14400, # seconds (4 hours) env={"PYTHONUNBUFFERED": "1"}, secrets={"HF_TOKEN": get_token()}, # MUST use get_token(), NOT "$HF_TOKEN" ) print(f"Job ID: {job_info.id}") ``` **Critical differences between the two methods:** | | `hf_jobs` MCP tool | `HfApi().run_uv_job()` | |---|---|---| | `script` param | Python code string or URL (NOT local paths) | File path to `.py` file (NOT content) | | Token in secrets | `"$HF_TOKEN"` (auto-replaced) | `get_token()` (actual token value) | | Timeout format | String (`"4h"`) | Seconds (`14400`) | **Rules for both methods:** - The training script MUST include PEP 723 inline metadata with dependencies - Do NOT use `image` or `command` parameters (those belong to `run_job()`, not `run_uv_job()`) ### 2. Authentication via job secrets + explicit hub_token injection **Job config** MUST include the token in secrets — syntax depends on submission method (see table above). **Training script requirement:** The Transformers `Trainer` calls `create_repo(token=self.args.hub_token)` during `__init__()` when `push_to_hub=True`. The training script MUST inject `HF_TOKEN` into `training_args.hub_token` AFTER parsing args but BEFORE creating the `Trainer`. The template `scripts/object_detection_training.py` already includes this: ```python hf_token = os.environ.get("HF_TOKEN") if training_args.push_to_hub and not training_args.hub_token: if hf_token: training_args.hub_token = hf_token ``` If you write a custom script, you MUST include this token injection before the `Trainer(...)` call. - Do NOT call `login()` in custom scripts unless replicating the full pattern from `scripts/object_detection_training.py` - Do NOT rely on implicit token resolution (`hub_token=None`) — unreliable in Jobs - See the `hugging-face-jobs` skill → *Token Usage Guide* for full details ### 3. JobInfo attribute Access the job identifier using `.id` (NOT `.job_id` or `.name` — these don't exist): ```python job_info = api.run_uv_job(...) # or hf_jobs("uv", {...}) job_id = job_info.id # Correct -- returns string like "687fb701029421ae5549d998" ``` ### 4. Required training flags and HfArgumentParser boolean syntax `scripts/object_detection_training.py` uses `HfArgumentParser` — all config is passed via `script_args`. Boolean arguments have two syntaxes: - **`bool` fields** (e.g., `push_to_hub`, `do_train`): Use as bare flags (`--push_to_hub`) or negate with `--no_` prefix (`--no_remove_unused_columns`) - **`Optional[bool]` fields** (e.g., `greater_is_better`): MUST pass explicit value (`--greater_is_better True`). Bare `--greater_is_better` causes `error: expected one argument` Required flags for object detection: ```