| name | omics-general |
| description | Cross-cutting principles for all omics analysis — AnnData/MuData universal containers, Python-first policy, data format detection, analysis phases, and shared analytical methods |
| version | 2.0.0 |
| tags | ["omics","bioinformatics","general-principles","anndata","mudata","python-first"] |
Omics General Principles
Cross-cutting guidance loaded by ALL sandbox agents.
Language Policy
Python is the default language. Use R when:
- No Python equivalent exists (e.g., minfi, ANCOM-BC2, ChAMP)
- The R implementation is significantly more mature (e.g., DESeq2 for complex contrasts)
Choose native R or rpy2 based on scope:
- Isolated R calls in a Python pipeline → use rpy2 bridge
- R-dominant pipeline (most steps are R packages) → write native R scripts
AnnData/MuData: Universal Data Containers
AnnData (.h5ad) is the universal container for ALL sample-by-feature data — not just single-cell:
| Modality | .obs (samples) | .var (features) | .X (values) |
|---|
| Bulk RNA-seq | samples, condition, batch | genes, ensembl_id, biotype | raw counts |
| Proteomics | samples, condition, batch | proteins, uniprot_id | intensities |
| Metabolomics | samples, condition, batch | metabolite features, mz, rt | abundances |
| Microarray | samples, condition, batch | probes/genes | normalized expression |
| Single-cell | cells, cell_type, condition | genes | counts or normalized |
| Methylation (results) | samples, condition, age | CpG sites, chr, pos | beta or M-values |
MuData (.h5mu) for multi-modal data: CITE-seq (RNA+protein), Multiome (RNA+ATAC), multi-omics integration.
AnnData Conventions
- Sample metadata →
.obs columns (never separate CSV files)
- Feature annotations →
.var columns
- Dimensionality reductions →
.obsm (e.g., X_pca, X_umap)
- Processed layers →
.layers (e.g., raw, normalized, log1p)
- Unstructured results →
.uns (DE results, method parameters)
- Save as
.h5ad / .h5mu — never .rds or .pkl
Converting from R Objects
When R produces objects (DESeqDataSet, phyloseq, SummarizedExperiment):
- Extract the data matrix, sample metadata, and feature metadata via rpy2
- Construct AnnData in Python from the extracted components
- Save as .h5ad for downstream steps
Data Format Detection
When receiving input data, detect the format before processing:
.h5ad → AnnData (load with anndata.read_h5ad).h5mu → MuData (load with mudata.read).csv/.tsv → tabular (load with pandas.read_csv, consider converting to AnnData).rds → R object (load via rpy2, convert to AnnData).h5/.hdf5 → HDF5 (inspect structure, may be AnnData).mtx + barcodes.tsv + features.tsv → 10x sparse (use scanpy.read_10x_mtx).loom → Loom format (use scanpy.read_loom, convert to AnnData)
Output Format Standards
- Intermediate data:
.h5ad or .h5mu (universal, travels with metadata)
- Tabular results: CSV with descriptive column names (
log2_fold_change, adjusted_pvalue)
- Figures: PNG (300 DPI) + PDF (vector) with same base filename
- Dual-format output: Save both native format AND generic CSV for cross-cutting agents
Analysis Plan Phases
- Data ingestion & QC — format detection, quality metrics, sample/feature filtering
- Preprocessing — normalization, batch correction, feature selection
- Core analysis — the primary analytical methods (DE, clustering, etc.)
- Downstream interpretation — pathway enrichment, regulatory inference, integration
- Visualization & reporting — publication-quality figures, summary tables
Key Cross-Cutting Methods
- Differential analysis: PyDESeq2 (bulk counts), limma via rpy2 (microarray/intensity), pseudobulk + PyDESeq2 (single-cell). Longitudinal: dream (variancePartition, R via rpy2)
- Pathway enrichment: decoupler with gene sets or pathway weights from the reference store; gseapy for GSEA/ORA
- TF activity: decoupler with a TF-target regulon network from the reference store
- Cell-cell communication: LIANA+
- GRN inference: pySCENIC — its motif-pruning step needs cisTarget reference data,
which is an opt-in download and may not be staged; check before planning around it
- Trajectory: scVelo, CellRank, Palantir
- Clustering: Leiden (preferred over Louvain)
- Batch correction: scVI (complex), Harmony (moderate), ComBat (bulk)
See references/general-principles.md for complete details.
Reference Data
Gene sets, regulons, pathway weights, and annotation resources are provisioned
as an optional reference store. Never assume a particular dataset is present
and never hardcode a path to one. Query the available references for what you
need, described by what the data is (e.g. "TF-target regulon network",
"hallmark gene sets"), and read the returned entry's stated format and column
layout before choosing a reader. Rules that hold regardless of which datasets
are installed:
- The store is read-only, and the sandbox has no network access — a
missing dataset cannot be downloaded or installed at runtime.
- If what you need is not there, report the gap and proceed with what is
available. Do not invent a path, and do not silently fall back to a
different organism's data.
- Match the reader to the declared format, not to the file extension:
.gmt gene sets are ragged (variable-length rows) — parse them
line-by-line, splitting on tabs. pandas.read_csv cannot read a GMT..rda / .rds are R serialized objects — load them through rpy2 (or in
native R), never with a pandas reader..csv / .tsv tables load with pandas; check whether the file has a
header, and whether it is gzipped or zipped, before reading.- Multi-species files must be filtered to the organism under analysis
before use — a human network silently runs over mouse counts.
Gene Annotation (R via rpy2)
When you need gene ID conversion, annotation, or organism databases:
- AnnotationDbi + org.Hs.eg.db / org.Mm.eg.db / org.Rn.eg.db / org.Dr.eg.db / org.Cf.eg.db / org.Bt.eg.db: Map between gene symbols, Entrez IDs, Ensembl IDs, UniProt IDs. Use
mapIds() or select().
- biomaRt: installed, but it queries Ensembl BioMart over the network and the sandbox has no outbound access — so
useMart() / getBM() will not connect. Do annotation offline instead: org.*.eg.db via AnnotationDbi (above), babelgene for orthologs, or the ID-mapping tables in the reference inventory. Reach for biomaRt only in an environment where network access is confirmed.
- ensembldb + EnsDb.Hsapiens.v86 / EnsDb.Mmusculus.v79: Gene models with coordinates, transcripts, exons. Use
genes(), transcripts().
- babelgene: Cross-species gene name conversion (human ↔ mouse ↔ rat). Use
orthologs().
R ↔ Python Format Conversion
- zellkonverter (R via rpy2): Convert between AnnData (.h5ad) and SingleCellExperiment (.rds). Use
readH5AD() / writeH5AD() in R when an R package requires SCE input.
Visualization Utilities
Python
- matplotlib + seaborn: Primary plotting (always available)
- plotly: Interactive plots. HTML export is built in; static PNG/SVG/PDF export goes through
kaleido (write_image)
- upsetplot: UpSet plots for set intersections (enrichment overlap, shared genes)
- matplotlib-venn: Venn diagrams (2-3 set comparisons)
- adjustText: Automatic label positioning to avoid overlap in scatter/volcano plots
- conorm: Count normalization utilities (CPM, TPM, RPKM/FPKM conversion)
R (via rpy2)
- ggplot2 + ggrepel: Publication-quality plots with non-overlapping labels
- pheatmap / ComplexHeatmap: Heatmaps with clustering dendrograms. ComplexHeatmap for advanced multi-track annotations.
- EnhancedVolcano: Publication-ready volcano plots with automatic labeling
- corrplot: Correlation matrix visualization
- patchwork / cowplot: Multi-panel figure layouts (combine ggplot panels)
- plotly (R): Interactive plots from R
