Query and download from NCBI Gene Expression Omnibus (GEO) and EMBL-EBI's BioStudies/ArrayExpress mirror. Use when finding expression datasets, navigating SuperSeries vs SubSeries, choosing between series-matrix (submitter-normalized) and raw supplementary files, downloading via GEOparse (Python) or GEOquery (R/Bioconductor), linking GEO to SRA for raw reads, or distinguishing GSE/GSM/GPL/GDS record types. Encodes the SuperSeries trap, the series-matrix normalization-trust caveat, GEOmetadb deprecation, ArrayExpress migration to BioStudies, and processed-vs-raw decision matrix.
Query and download from NCBI Gene Expression Omnibus (GEO) and EMBL-EBI's BioStudies/ArrayExpress mirror. Use when finding expression datasets, navigating SuperSeries vs SubSeries, choosing between series-matrix (submitter-normalized) and raw supplementary files, downloading via GEOparse (Python) or GEOquery (R/Bioconductor), linking GEO to SRA for raw reads, or distinguishing GSE/GSM/GPL/GDS record types. Encodes the SuperSeries trap, the series-matrix normalization-trust caveat, GEOmetadb deprecation, ArrayExpress migration to BioStudies, and processed-vs-raw decision matrix.
Before using code patterns, verify installed versions match. If versions differ:
Python: pip show biopython geoparse then introspect signatures
R: packageVersion('GEOquery')
If the GSE structure doesn't match expectations (missing fields, malformed series matrix), re-fetch from FTP directly and inspect the SOFT or MINiML file as source of truth.
GEO Data
"Pull expression data from GEO accession GSE..." -> GEO stores Series (GSE), Samples (GSM), Platforms (GPL), and curated DataSets (GDS, frozen 2018). The single most consequential decision is processed (series matrix) vs raw (supplementary files / linked SRA) — the answer turns on how much trust the submitter's normalization deserves.
The single most-missed gotcha: SuperSeries. A GSE may be a meta-container (!Series_relation = SuperSeries of: GSExxxxx) holding multiple sub-studies on different platforms. Naively pulling samples from a SuperSeries gives mixed Affymetrix + Illumina + RNA-seq, mis-batched.
Python: Entrez.esearch(db='gds'), GEOparse for full series download
R: GEOquery::getGEO() (Bioconductor; more mature than GEOparse)
CLI: wget from ftp.ncbi.nlm.nih.gov/geo/series/...
Required Setup
pip install biopython GEOparse pandas
# OR for R-side:# R: BiocManager::install('GEOquery')
from Bio import Entrez
Entrez.email = 'researcher@institution.edu'
Entrez.api_key = 'optional'
GEO record taxonomy
Prefix
Type
Granularity
What's in it
GSE
Series
One study
Title, summary, design, links to GSMs, supplementary files
GSM
Sample
One biological/technical sample
Submitter metadata, per-sample processed data, link to raw SRA
GPL
Platform
One array / sequencer
Probe annotations or sequencer model
GDS
DataSet
Curated, normalized subset of one GSE
Re-normalized expression matrix (frozen 2018; new GDS no longer created)
GSEXXX SuperSeries
Series meta-container
Wraps multiple SubSeries
!Series_relation = SuperSeries of: ...
GDS is dead-as-format: NCBI stopped creating new GDS records in 2018. Existing GDS still queryable but use GSE for anything current.
The SuperSeries trap
A SuperSeries (GSE) wraps multiple SubSeries, often with different platforms. Detection:
# Read the !Series_relation field from SOFT formatfrom Bio import Entrez
h = Entrez.esummary(db='gds', id='200122288') # example
r = Entrez.read(h)[0]; h.close()
print(r.get('summary')) # may or may not flag SuperSeries# Definitive check: download SOFT and grep:# curl ftp://ftp.ncbi.nlm.nih.gov/geo/series/GSE122nnn/GSE122288/soft/GSE122288_family.soft.gz | zgrep Series_relation
A SuperSeries of: GSE12345 line means the SuperSeries' samples are the union of all SubSeries — almost certainly mixed-platform / mixed-batch. Process each SubSeries independently.
Symmetric trap: a paper may cite a SubSeries (SubSeries of: GSEsuper) where the wider context is essential — check both directions.
Decision matrix: processed vs raw vs SRA
Question
Source
Trust level
"I want expression values; submitter normalization is fine"
Series matrix (GSE_series_matrix.txt.gz)
Trust submitter's normalization
"I want raw Affymetrix CEL files and to do my own RMA"
Default to raw whenever possible. For Affymetrix: CEL + locally-run RMA is far more reliable than the submitter's "normalized" matrix. For RNA-seq: SRA FASTQ + locally-run alignment/quantification is the only reproducible path; submitter counts often use a private pipeline.
Series matrix files
A series matrix (GSE12345_series_matrix.txt.gz) is a header (sample metadata as !Sample_* lines) plus a sample-by-feature expression table. The format is fragile and the values' provenance is whatever the submitter chose. Critical caveats:
For Affymetrix: the matrix is usually RMA-normalized but submitters sometimes apply additional transforms (log2, scaling, batch correction).
For RNA-seq: the matrix is sometimes log-CPM, sometimes raw counts, sometimes VST/rlog — read !Series_overall_design and !Sample_data_processing to know.
The header has !Sample_characteristics_ch1 rows that hold the metadata of interest — these are submitter-formatted strings, often inconsistent within one series.
SOFT vs MINiML
Format
Content
Parser support
SOFT (*_family.soft.gz)
Plain-text, key=value style
GEOparse (Python), GEOquery (R), Entrez Direct
MINiML (*_family.xml.tgz)
XML-structured
GEOparse, GEOquery, custom XML
Both contain the same content. SOFT is the legacy, MINiML the XML successor. GEOparse handles SOFT well; for very large series (1000+ samples) MINiML's XML structure is slower to parse.
GEOparse vs GEOquery
Aspect
GEOparse (Python)
GEOquery (R/Bioconductor)
Maturity
OK; some known supplementary-file fetch issues since ~2022
Mature; Bioconductor-supported
Output
GEOparse.GSE object with gsms, gpls, metadata dicts
For production GEO workflows in R, GEOquery is the stable choice. For Python, GEOparse is the only option but verify file counts after download.
GEOmetadb status
GEOmetadb (Zhu 2008) was a SQLite mirror of GEO metadata enabling fast SQL queries. Unmaintained since 2020; downloads still work but data is stale. Modern replacement: pysradb (pysradb gse_to_srp, pysradb metadata) covers most of the GEO->SRA mapping; for full GEO queries fall back to Entrez gds.
ArrayExpress -> BioStudies migration (2020)
ArrayExpress (EMBL-EBI's microarray archive, mirroring GEO) was migrated into BioStudies in 2020. Old E-MTAB-#### accessions still resolve but the API moved:
Fix: Switch to pysradb for SRA-linked queries; Entrez gds for full GEO.
Common errors
Error / symptom
Cause
Solution
Empty IdList for gse[entry_type]
Wrong field name
Use gse[Entry Type] (case-sensitive)
Matrix file has no expression data
SuperSeries with no aggregate matrix
Pull per-SubSeries matrices
Submitter "normalized" matrix gives different result than paper
Hidden submitter transforms
Re-process from raw
404 on ArrayExpress URL
Migrated to BioStudies
Use new BioStudies URL
GEOparse missing CEL files
Known flake
Use R GEOquery or direct FTP
GEOmetadb-based pipeline missing recent series
DB unmaintained
Switch to pysradb / Entrez
References
Edgar R, Domrachev M, Lash AE. (2002) Gene Expression Omnibus: NCBI gene expression and hybridization array data repository. Nucleic Acids Res 30:207-210.
Barrett T, Wilhite SE, Ledoux P, et al. (2013) NCBI GEO: archive for functional genomics data sets - update. Nucleic Acids Res 41:D991-D995.
Davis S, Meltzer PS. (2007) GEOquery: a bridge between the Gene Expression Omnibus (GEO) and BioConductor. Bioinformatics 23:1846-1847.
Sarkans U, Gostev M, Athar A, et al. (2018) The BioStudies database--one stop shop for all data supporting a life sciences study. Nucleic Acids Res 46:D1266-D1270.
Lachmann A, Torre D, Keenan AB, et al. (2018) Massive mining of publicly available RNA-seq data from human and mouse. Nat Commun 9:1366. (ARCHS4)
Wilks C, Zheng SC, Chen FY, et al. (2021) recount3: summaries and queries for large-scale RNA-seq expression and splicing. Genome Biol 22:323.
Related Skills
entrez-search - General gds search
entrez-link - gds <-> pubmed, bioproject links (gds->sra ELink is unreliable; use pysradb)
sra-data - Download raw FASTQ from GEO-linked SRA runs
expression-matrix/normalization - Re-normalize raw expression data
rna-quantification/alignment-free-quant - Salmon/kallisto re-quantification of GEO/SRA data
ensembl-rest - Cross-reference Ensembl IDs in series-matrix files
