Download genome assemblies, gene records, and ortholog data from NCBI using the modern Datasets v2 CLI (replaces assembly_summary.txt scraping and many EFetch workflows). Use when bulk-pulling genome assemblies, gene metadata across species, ortholog sets, or BLAST databases; when E-utilities are too slow for genome-scale work; or when automatic checksum verification, parallel download, and clean accession-driven retrieval are required. Encodes the JSON-lines output format, dataformat conversion, --dehydrated for cloud workflows, and when Datasets is/isn't the right tool.
Instalar com Codex ou Claude Copie este prompt, cole no Codex, Claude ou outro assistente e deixe que ele revise a página da skill e instale para você.
Um comando direto ignora o prompt de revisão. Verifique a origem antes de executá-lo.
Download genome assemblies, gene records, and ortholog data from NCBI using the modern Datasets v2 CLI (replaces assembly_summary.txt scraping and many EFetch workflows). Use when bulk-pulling genome assemblies, gene metadata across species, ortholog sets, or BLAST databases; when E-utilities are too slow for genome-scale work; or when automatic checksum verification, parallel download, and clean accession-driven retrieval are required. Encodes the JSON-lines output format, dataformat conversion, --dehydrated for cloud workflows, and when Datasets is/isn't the right tool.
Before using code patterns, verify installed versions match. If versions differ:
CLI: datasets --version, dataformat --version
Subcommand help: datasets <subcommand> --help
If a subcommand or flag is unrecognized, run datasets --help and adapt. The CLI is under active development; major releases (v15 -> v16) added subcommands and renamed flags.
NCBI Datasets CLI
"Pull genome / gene / ortholog data from NCBI in 2026" -> The Datasets v2 CLI (launched 2023) is the official, supported bulk endpoint for genome and gene-centric data. It replaces the prior best-practice of scraping assembly_summary.txt + parallel FTP + manual checksum verification. For genome-scale data, it is strictly better than E-utilities (EFetch).
The CLI is not the right answer for everything. PubMed, SRA reads, and custom Entrez queries still belong to E-utilities. The defection rule: if the question is about genome assemblies, gene records, or pre-computed orthologs, use Datasets; otherwise stay with E-utilities.
CLI: datasets download genome accession GCF_...
CLI: datasets summary gene symbol BRCA1 --taxon human
Python: subprocess wrapper; Python client ncbi-datasets-pylib (experimental as of 2024)
Installation
# conda
conda install -c conda-forge ncbi-datasets-cli
# Or direct download (Linux, macOS, Windows binaries)
curl -O https://ftp.ncbi.nlm.nih.gov/pub/datasets/command-line/v2/linux-amd64/datasets
datasets --version # 16.0+ expected
dataformat --version # bundled companion tool
What's in scope (use Datasets) vs out of scope (use E-utilities or other tools)
datasets summary always returns JSON-lines on stdout (one object per record). datasets download produces a .zip (default) or a "dehydrated" stub for cloud workflows.
Restrict to reference assemblies only (one per species)
--annotated
Restrict to annotated assemblies
--assembly-source RefSeq / GenBank / all
Database source
--assembly-level chromosome,complete
Assembly quality level
--released-after 2024-01-01
Date filter
--dehydrated
Skip data; download just stubs + URL list (for parallel pull)
--api-key XXX
Optional API key (raises rate limit)
--no-progressbar
For non-interactive use
For very large pulls (1000+ genomes), --dehydrated is the right choice: download the metadata stubs first, then run datasets rehydrate later or pull URLs in parallel from the manifest.
JSON-lines output + dataformat
datasets summary returns JSON-lines (one JSON object per line) on stdout. Pipe through dataformat tsv for tabular:
dataformat subcommands match summary types: genome, gene, virus-genome, etc. The --fields list is documented per type via dataformat tsv <type> --help.
When to use --dehydrated for cloud workflows
The "dehydrated" mode separates data discovery from data transfer:
Inspect: unzip -p human.zip ncbi_dataset/fetch.txt -- a TSV of all URLs to pull.
Pull: either datasets rehydrate --directory ./human/ or use aria2c --input-file=fetch.txt for parallel pull.
This is essential for HPC / cloud pipelines where inspection of the pending transfer is needed before committing the I/O.
Checksum verification (automatic)
datasets verifies MD5 checksums for every downloaded file automatically. Rehydrate workflows also verify. If a file fails checksum, Datasets retries up to 3 times then errors. This replaces the md5sum -c step that was required with assembly_summary.txt-based scraping.
Code patterns
Download a single reference genome
Goal: Get human reference assembly with genome + GTF + protein + CDS.
datasets summary gene symbol BRCA1 --taxon human --ortholog --as-json-lines \
| dataformat tsv gene --fields gene-id,symbol,taxname,description \
> brca1_orthologs.tsv
--ortholog returns NCBI's ortholog set (a single representative per species; tree-aware orthology with multiple co-orthologs is in ortholog-inference / Compara / OMA).
O'Leary NA, Cox E, Holmes JB, et al. (2024) Exploring and retrieving sequence and metadata for species across the tree of life with NCBI Datasets. Sci Data 11:732.
