Download large datasets from NCBI efficiently using EPost, history server, batching, rate limiting, and retry logic. Use when bulk-fetching tens of thousands of sequences, pulling all results of a large ESearch, designing reproducible pipelines, comparing E-utilities to NCBI Datasets v2 CLI, or implementing checksum-validated downloads. Encodes WebEnv TTL (~8h), EPost 200-ID limit, retmax caps, parallelization design, and integrity verification.
Download large datasets from NCBI efficiently using EPost, history server, batching, rate limiting, and retry logic. Use when bulk-fetching tens of thousands of sequences, pulling all results of a large ESearch, designing reproducible pipelines, comparing E-utilities to NCBI Datasets v2 CLI, or implementing checksum-validated downloads. Encodes WebEnv TTL (~8h), EPost 200-ID limit, retmax caps, parallelization design, and integrity verification.
Before using code patterns, verify installed versions match. If versions differ:
Python: pip show biopython then help(Bio.Entrez.efetch) to check signatures
CLI: datasets --version and efetch -version
If code throws ImportError, AttributeError, or TypeError, introspect the installed
package and adapt the example to match the actual API rather than retrying.
Batch Downloads
"Download N thousand records from NCBI without getting blocked" -> The right answer is rarely "parallelize requests". For >5000 records the answer is the history server: search once, fetch in chunks server-side. For >100,000 records or whole genomes, the modern answer is NCBI Datasets v2 CLI -- the E-utilities are not optimized for bulk genome/gene data anymore.
This skill encodes (a) when to use each retrieval strategy, (b) the precise rate-limit math, (c) WebEnv lifecycle for long-running jobs, (d) how to design retry/resume, and (e) when to defect to Datasets CLI instead.
from Bio import Entrez
import time
Entrez.email = 'researcher@institution.edu'
Entrez.api_key = 'YOUR_KEY'# 3 -> 10 req/sec; mandatory for bulk
Entrez.tool = 'project-name'
Decision matrix: which retrieval strategy?
Record count
Source
Strategy
Why
< 200 known IDs
Any db
EFetch with comma-joined id=
Single round-trip; trivial
200-5,000 known IDs
Any db
EPost (chunked at 200) -> history -> chunked EFetch
URL length limit + chunked retrieval
5,000-100,000 from a query
Any db
ESearch with usehistory='y' -> chunked EFetch
Push to server once; pull in batches
> 100,000 sequences
nucleotide/protein
Consider FTP mirror or Datasets CLI; chunk if E-utils still
NCBI throttles bulk; offline mirror is faster
Whole genome assemblies
Assembly/Datasets
datasets download genome accession ...
Datasets v2 is the modern bulk endpoint
All RefSeq for a species
Datasets
datasets download genome taxon ...
Replaces assembly_summary.txt scraping
All gene records for a list
Datasets
datasets download gene gene-id ...
Cleaner output than EFetch gene XML
Raw sequencing reads
SRA
prefetch + fasterq-dump (or ENA mirror)
See sra-data skill
The Datasets CLI is the right answer for any genome- or gene-centric bulk workflow as of 2023+. The E-utilities remain right for PubMed, ESummary metadata, custom queries, and anything not in the Datasets API. See ncbi-datasets-cli skill.
Rate-limit math (precise)
Auth
req/sec
Sleep between calls
Bulk-friendly notes
Email only
3
0.34 s
Single-threaded only; parallelism violates ToS
Email + API key
10
0.10 s
Modest parallelism (max ~4 workers) safe
Institutional bulk
Negotiated
Email eutilities@ncbi.nlm.nih.gov
For >100K queries; courtesy expected
NCBI's terms ask that heavy automated downloads run outside US weekday business hours (9 AM-5 PM ET). Cron the job for nights/weekends; pipelines that ignore this get IP-throttled.
Critical: parallelizing API calls is the WRONG bulk strategy. One stream with history server + larger batches is faster AND more polite than N parallel streams. The bottleneck is rarely NCBI's throughput at small N -- it's the round-trip count.
History server lifecycle (the long-running-job trap)
Property
Value
Failure mode
TTL
8 hours absolute (per NCBI E-utils help)
Job started Friday evening dies Saturday morning
Idle eviction
~15 min empirically under load
A worker that stalls loses its WebEnv
Per-session isolation
One WebEnv string per session
Don't share across processes if isolation matters
Expired session behavior
HTTP 200 with <ERROR>WebEnv not found</ERROR>
Won't surface as HTTP error -- must parse body
Recovery
Re-run ESearch; resume at retstart
Need to checkpoint progress to disk
Production pattern: checkpoint the retstart cursor after each successful chunk to disk; on restart, re-run ESearch (cheap), pick up retstart from checkpoint, continue.
EPost specifics
EPost pushes a list of UIDs to the history server so downstream EFetch can pull by WebEnv/QueryKey instead of by ID. Two constraints:
200 IDs per EPost call is the hard limit.
Chained posts share a WebEnv: pass the WebEnv from the first call into subsequent calls to accumulate IDs under one session; a new QueryKey is issued per call.
To intersect: term=#{key1} AND #{key2} against the WebEnv produces a new key.
Batch size guidelines per rettype
Database
rettype
Optimal batch
Per-record payload
nucleotide
fasta
500-1000
~1 KB
nucleotide
gb
100-200
~10-50 KB
protein
fasta
500-1000
~0.5 KB
protein
gp
100-200
~5-30 KB
pubmed
medline
1000-2000
~2 KB
pubmed
xml
200-500
~10-30 KB
any
esummary (docsum)
500 per call
~1 KB
Smaller batches for GenBank/XML because per-record payload is larger; larger batches for FASTA because the per-call HTTP overhead dominates.
Code patterns
Production batch fetch (history server + retry + checkpoint)
Goal: Download all records matching a query, robust to mid-job failures and session expiry.
Approach: ESearch with history; checkpoint cursor to disk; on error, retry the chunk; on session expiry, re-run ESearch and resume from checkpoint.
Reference (BioPython 1.83+):
import json
import time
from pathlib import Path
from urllib.error import HTTPError
from Bio import Entrez
defcheckpointed_batch_download(db, term, out_path, ckpt_path, rettype='fasta',
retmode='text', batch_size=500, max_retries=3):
'''Download all matching records with disk checkpoint for resumability.'''
delay = 0.1if Entrez.api_key else0.34
ckpt = Path(ckpt_path)
start = json.loads(ckpt.read_text())['start'] if ckpt.exists() else0
h = Entrez.esearch(db=db, term=term, usehistory='y', retmax=0)
s = Entrez.read(h); h.close()
webenv, query_key, total = s['WebEnv'], s['QueryKey'], int(s['Count'])
print(f'{total:,} records matched; resuming at {start:,}')
mode = 'a'if start else'w'withopen(out_path, mode) as out:
while start < total:
for attempt inrange(max_retries):
try:
h = Entrez.efetch(db=db, rettype=rettype, retmode=retmode,
retstart=start, retmax=batch_size,
webenv=webenv, query_key=query_key)
body = h.read(); h.close()
ifisinstance(body, bytes):
body = body.decode('utf-8', errors='replace')
if'<ERROR>'in body[:500]:
raise RuntimeError(f'Server error in body: {body[:200]}')
out.write(body)
breakexcept HTTPError as e:
if e.code == 429:
wait = 10 * (attempt + 1)
print(f' Rate-limited; sleeping {wait}s')
time.sleep(wait)
elif attempt == max_retries - 1:
raiseelse:
time.sleep(5 * (attempt + 1))
except RuntimeError as e:
# Likely WebEnv expired; re-run ESearchprint(f' {e}; refreshing WebEnv')
h = Entrez.esearch(db=db, term=term, usehistory='y', retmax=0)
s = Entrez.read(h); h.close()
webenv, query_key = s['WebEnv'], s['QueryKey']
start += batch_size
ckpt.write_text(json.dumps({'start': start, 'total': total}))
time.sleep(delay)
print(f' {min(start, total):,}/{total:,}')
ckpt.unlink(missing_ok=True)
EPost large ID list, then EFetch
Goal: Download by a known list of 5,000 accessions without 414 URI errors.
Approach: EPost in 200-ID chunks; reuse WebEnv across chunks; final fetch reads from history.
Reference (BioPython 1.83+):
defepost_and_fetch(db, ids, out_path, rettype='fasta', retmode='text', batch_size=500):
delay = 0.1if Entrez.api_key else0.34
webenv = None
posted_keys = [] # (query_key, n_ids) so we iterate each key's actual sizefor i inrange(0, len(ids), 200):
chunk = ids[i:i+200]
kwargs = {'db': db, 'id': ','.join(chunk)}
if webenv:
kwargs['WebEnv'] = webenv
h = Entrez.epost(**kwargs)
r = Entrez.read(h); h.close()
webenv = r['WebEnv']
posted_keys.append((r['QueryKey'], len(chunk)))
time.sleep(delay)
withopen(out_path, 'w') as out:
for qk, n in posted_keys:
for start inrange(0, n, batch_size):
h = Entrez.efetch(db=db, rettype=rettype, retmode=retmode,
retstart=start, retmax=min(batch_size, n - start),
webenv=webenv, query_key=qk)
out.write(h.read()); h.close()
time.sleep(delay)
Integrity check after download
Goal: Confirm downloaded FASTA has the expected record count and no truncation.
Approach: Count expected (from ESearch Count) vs observed (from SeqIO.parse).
from Bio import SeqIO
defverify_fasta_count(path, expected):
observed = sum(1for _ in SeqIO.parse(path, 'fasta'))
assert observed == expected, f'Expected {expected:,} records, found {observed:,}'returnTrue
For genome assemblies and known-checksum files, NCBI provides MD5 manifests (e.g. md5checksums.txt in FTP genome directories). NCBI Datasets CLI verifies checksums automatically; the FTP-direct route needs explicit md5sum -c.
For 100,000 nucleotide records at 500/batch with API key: 200 calls * 0.1s = 20s minimum. For the same workflow via datasets download gene gene-id 100000: one CLI invocation, parallel download, automatic checksum. For genome-scale bulk, Datasets wins by an order of magnitude.
Parallelization design (modest)
Goal: Pull from two independent queries concurrently without violating rate limits.
Approach: Async with a global semaphore that enforces the API-key-permitted rate. Max 4 concurrent workers is the polite cap.
import asyncio
from asyncio import Semaphore
# Pseudo-pattern; real impl needs aiohttp + Bio.Entrez async wrappersasyncdeffetch_with_semaphore(sem, db, id_, rettype):
asyncwith sem:
# call EFetchawait asyncio.sleep(0.1) # rate gate# ... actual call
sem = Semaphore(4)
Never exceed 4 concurrent workers with an API key, or 1 without. Above that NCBI throttles by IP and the whole pipeline grinds.
Failure modes
Session expires mid-pipeline
Trigger: Job runs >8h or worker idles >15 min.
Mechanism: WebEnv evicted; EFetch returns HTTP 200 with <ERROR>WebEnv not found</ERROR> body.
