InternVideo2-CLIP L14 (TAO video_clip) for video-text retrieval, zero-shot classification, embedding extraction, LoRA fine-tuning, and ONNX export. Use when the user asks to "fine-tune IV2CLIP", "run video_clip train/evaluate/inference/export", "InternVideo2-CLIP on KPI chunks", or "TAO video_clip on vadr1_chunks JSON".
Instrucciones de origen · Vista previa de solo lectura
name
tao-finetune-video-clip
description
InternVideo2-CLIP L14 (TAO video_clip) for video-text retrieval, zero-shot classification, embedding extraction, LoRA fine-tuning, and ONNX export. Use when the user asks to "fine-tune IV2CLIP", "run video_clip train/evaluate/inference/export", "InternVideo2-CLIP on KPI chunks", or "TAO video_clip on vadr1_chunks JSON".
license
Apache-2.0
compatibility
Requires docker + nvidia-container-toolkit and the pinned TAO video_clip container (see references/skill_info.yaml), or a local tao-pytorch checkout + tao-cli venv for virtualenv runs. MobileCLIP + InternVideo2 weights must be on disk for offline eval (HF LFS may be blocked in CI). Metadata JSON uses vadr1_chunks with absolute video_path entries.
Standalone install? If this session was not initialized by the TAO skill bank plugin, run the tao-setup skill first (host preflight, credentials, cross-skill discovery).
TAO task video_clip wraps OpenGVLab InternVideo2-CLIP L14. The CLI entrypoint is video_clip with actions train, evaluate, inference, export, and default_specs.
Container image and per-action commands are in references/skill_info.yaml. Starting specs are in references/spec_template_*.yaml.
Release note: The pinned image is an interim nvstaging build until tao-pytorch !620 merges and TAO publishes an official video_clip image tag. After release, versions.yamlimages.tao_toolkit.video_clip will be re-stamped.
Known-broken images: interim builds cut before tao-pytorch commit 0cc31de4 (v7.0.1-pyt2.1.0-py3-01 and -py3-02, both superseded by the pinned -py3-03) ship a video_clip package with no model.backbones submodule, so train/evaluate/inference all die at import with ModuleNotFoundError: nvidia_tao_pytorch.multimodal.video_clip.model.backbones — while video_clip --help still exits 0. Run the import smoke check in the preflight below before pulling data or launching a run.
Train Action Policy
AutoML is not packaged for this model skill. Always use direct video_clip actions even when a higher-level request mentions AutoML. Non-train actions stay in this skill.
Quick Start (local Docker)
Use the pinned TAO container declared in references/skill_info.yaml. Pull with NGC_KEY when the image is not cached locally.
[ -f "$RUN_ROOT/model/mobileclip_blt.pt" ] || echo"MISSING: MobileCLIP weights"
[ -f "$RUN_ROOT/data/train.json" ] || echo"MISSING: train metadata"
[ -d "$RUN_ROOT/model/hf" ] || echo"MISSING: offline HF snapshot under model/hf"
docker run --rm"$VIDEO_CLIP_IMAGE" video_clip --help >/dev/null || echo"MISSING: video_clip in container"
docker run --rm"$VIDEO_CLIP_IMAGE" \
python -c "import nvidia_tao_pytorch.multimodal.video_clip.model.adapters.internvideo2clip" \
>/dev/null 2>&1 || echo"BROKEN IMAGE: video_clip package is incomplete (missing model.backbones) — stop, see Release note"
docker run --rm"$VIDEO_CLIP_IMAGE" python -c "import decord" \
>/dev/null 2>&1 || echo"NO DECODE BACKEND: decord is not in the image and the container ffmpeg/opencv are codec-disabled — build a derived image (FROM \$VIDEO_CLIP_IMAGE + RUN pip install decord==0.6.0) and run every action from it"
nvidia-smi >/dev/null 2>&1 || echo"note: no GPU visible"
Train:
docker run "${DOCKER_COMMON[@]}""$VIDEO_CLIP_IMAGE" \
video_clip train -e /specs/train.yaml results_dir=/results
Evaluate:
docker run "${DOCKER_COMMON[@]}""$VIDEO_CLIP_IMAGE" \
video_clip evaluate -e /specs/evaluate.yaml results_dir=/results
Inference:
docker run "${DOCKER_COMMON[@]}""$VIDEO_CLIP_IMAGE" \
video_clip inference -e /specs/inference.yaml results_dir=/results
Export:
docker run "${DOCKER_COMMON[@]}""$VIDEO_CLIP_IMAGE" \
video_clip export -e /specs/export.yaml results_dir=/results
Quick Start (virtualenv — local dev hosts)
On hosts with a tao-pytorch checkout and tao-cli venv (for example rtdetr-pytorch), run through tao-run-on-virtualenv instead of Docker:
export VENV="${VENV:?set to your tao-cli virtualenv (must contain bin/video_clip)}"export TAO_PYTORCH_ROOT="${TAO_PYTORCH_ROOT:?set to your tao-pytorch checkout with multimodal/video_clip}"export PATH="$VENV/bin:$PATH"export PYTHONPATH="$TAO_PYTORCH_ROOT:$TAO_PYTORCH_ROOT/tao-core:$PYTHONPATH"export HF_HOME="${HF_HOME:-$PWD/hf_cache}"export HF_HUB_OFFLINE="${HF_HUB_OFFLINE:-1}"export WANDB_DISABLED=trueexport WANDB_MODE=disabled
Copy a template spec from references/spec_template_*.yaml, fill checkpoint paths and metadata, then run:
NGC_KEY — pull the pinned TAO container from nvcr.io when it is not cached locally.
HF_TOKEN (only when weights are not already cached on the host):
HuggingFace read token used to resolve the InternVideo2 snapshot named by
model.internvideo2clip_hf_id. With HF_HUB_OFFLINE=1 and local MobileCLIP +
InternVideo2 weights already on disk, no token is required.
Treat tokens as secrets. Export them into the environment or pass them through an
--env-file of bare KEY=value lines, rather than inlining values into generated
spec YAML, command lines, or anything written under results_dir.
Data format (vadr1_chunks)
Metadata is a top-level JSON list of video records. Each record has video_path, split, and nested chunks[] with caption fields (queries, action_queries, anomaly_queries, dense_caption, scene_caption).
Point dataset.*.video_text.metadata at the user's JSON files (for example /data/train.json and /data/val.json in the templates). Each video_path must be an absolute path resolvable inside the runtime — for Docker, remap host clips to container paths like /data/videos/<name>.mp4 and set data_root: null unless using path_prefix_mapping.
Smoke overrides
For a short functional check (for example 2 epochs, 1 GPU, small batch):
Use dataset.metrics.mode: retrieval only when dataset.val.video_text.relevance_file is provided.
Inference
inference.mode: embeddings writes video_embeddings.h5 and text_embeddings.h5 under results_dir.
Provide inference.query.text_file (one prompt per line) and/or inference.query.input_texts.
Gallery videos come from dataset.inference.video_text.metadata.
Export
export.encoder_type: separate writes vision/text ONNX under export.onnx_file (base path). Default opset is 23 on the vendor branch. Requires a trained .pth at export.checkpoint.
LoRA
For vision-LoRA runs, start from tao-pytorchexperiment_spec_lora.yaml or add a top-level peft: block (see shipped spec comments). Merge LoRA before export when checkpoints contain lora_* keys.
Common pitfalls
PATH must prefer $VENV/bin on virtualenv hosts so child processes resolve the venv Python.
evaluate uses dataset.val, not a separate test split — Lightning stage "test" still loads val metadata.
Eval precision: match train.precision (typically bf16) or flash-attn paths may fail under fp32 eval.
Empty action_queries on normal chunks become literal "Normal" positives during training; exclude Normal/Abnormal in dataset.metrics.exclude_categories for classification eval.
No hard-negative / explicit-neg training on the vendor branch unless the spec and branch explicitly enable it.
video_clip --help is not a health check. It exits 0 on an image whose video_clip package is missing model.backbones; only the import smoke check in the preflight catches it.
The pinned image has no working video-decode backend.decord is not installed, the bundled ffmpeg is codec-disabled (no pipe protocol, no rawvideo muxer, no select filter), and the opencv build has WITH_FFMPEG=OFF, so train dies at data-loading with All video decoding backends failed. Until the base image is rebuilt from tao-pytorch !620 — which pins decord==0.6.0 in docker/requirements-pip.txt — build a derived image (FROM $VIDEO_CLIP_IMAGE + RUN pip install decord==0.6.0) and run every action from it. A wheel-only release rebuild does not fix this: release/docker/Dockerfile consumes the base by hardcoded digest, so base-level dependency pins only land after a base-image rebuild.
PyTorch ≥ 2.6 defaults to torch.load(weights_only=True) and rejects the TAO checkpoint’s numpy dtype objects with _pickle.UnpicklingError. TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD=1 is set in DOCKER_COMMON above; keep it for evaluate, inference, and export.
model/hf/ is a snapshot, not an HF hub cache. Offline packs stage InternVideo2 weights at repo-relative paths (stage1/L14/L14_dist_1B_stage2/pytorch_model.bin, clip/L14/pytorch_model.bin), while HUGGINGFACE_HUB_CACHE expects a models--<org>--<repo>/snapshots/<sha>/ tree. Leaving model.vision_encoder / model.clip_head at null sends asset resolution to hf_hub_download and fails under HF_HUB_OFFLINE=1 — point both at the files directly.
CI / skill-eval: stage weights + remapped JSON under $WORKSPACE_DIR from S3; do not rely on HuggingFace LFS downloads at eval time.
References
Shipped defaults: tao-pytorchnvidia_tao_pytorch/multimodal/video_clip/experiment_specs/ (when developing from source)
