dvc

name	dvc
description	Data Version Control (DVC) for tracking large datasets/models with Git-like semantics, defining reproducible data/ML pipelines (dvc.yaml stages that only re-run when their inputs change), and lightweight experiment tracking without a server. Use when large files (VCF/BAM/FASTQ, reference genomes, model weights) can't go in Git, when you need Make/Snakemake-style selective re-execution driven by data, or when comparing many training runs locally before promoting one. Pairs with Git (code), cloud object storage (data), and Snakemake/Nextflow (compute graph).
license	Apache-2.0
allowed-tools	["Read","Write","Edit","Bash"]
compatibility	Requires Python 3.9+ and dvc 3.x (current 3.67.1). Remote storage needs a matching extra (dvc[s3], dvc[gs], dvc[azure], dvc[ssh], ...) so the storage SDK is installed alongside the core CLI. The upstream GitHub org moved from iterative/dvc to treeverse/dvc in early 2026 after Treeverse (the lakeFS company) acquired Iterative; the PyPI package name (`dvc`), CLI, and file formats are unchanged.
metadata	{"version":"1.0","skill-author":"community"}

DVC (Data Version Control)

Overview

DVC extends Git to the parts Git is bad at: large binary files and reproducible pipelines. Git tracks a small .dvc pointer file (a content hash); DVC stores the actual data in a local cache and, optionally, a remote (S3, GCS, Azure, SSH, or a plain directory). On top of that, DVC provides a lightweight pipeline definition (dvc.yaml) that re-runs only the stages whose inputs changed, and an experiment tracker (dvc exp) that snapshots code+data+params+metrics for each run without needing a tracking server.

Think of it as: Git manages the recipe (code), DVC manages the ingredients (data, models) and can also manage the cooking steps (pipeline stages) and taste-tests (experiments).

DVC is not a replacement for Snakemake/Nextflow when you need complex scheduling, retries, or cluster execution — it complements them. A common pattern is DVC tracking pipeline inputs/outputs while Snakemake or Nextflow drives the actual multi-node execution.

Installation

# Core CLI, local/directory remotes only
uv pip install dvc

# Pick the extra(s) matching your remote storage backend
uv pip install "dvc[s3]"       # AWS S3 (boto3)
uv pip install "dvc[gs]"       # Google Cloud Storage
uv pip install "dvc[azure]"    # Azure Blob Storage
uv pip install "dvc[ssh]"      # SSH/SFTP remote
uv pip install "dvc[all]"      # every backend (heavier install)

# DVCLive for in-training metric logging (separate package)
uv pip install dvclive

Check version: dvc --version (targets 3.67.x; commands below are stable across the 3.x series).

When to Use

• A dataset, model checkpoint, or reference genome is too large for Git (VCF, BAM, FASTQ, .pt/.safetensors weights, indexed references).
• You want make/snakemake-style "only re-run what changed" behavior driven by data hashes, not just file mtimes.
• You need to compare dozens of training runs (params, metrics, code diff) without standing up an MLflow/W&B server.
• You want data lineage that lives in the same Git history as the code that produced it (auditable, PR-reviewable).