forge

# FORGE on a SLURM cluster A practical guide to running FORGE (Flow Orchestrated Regulatory Genomics Engine) on a shared SLURM cluster with **no root access**, driven from Operon over SSH. Compiled 2026-09-06 against https://github.com/swaruplabUCI/FORGE (README, `docs/`, `configs/`, `launch.sh`, `launch_tutorial.sh`, `hpc_defs/BUILD_ON_HPC.sh`, `main.nf`, `nextflow.config`). Everything below comes from those files; where the repository says something is "to be provided" or unreleased, this guide says so rather than guessing. Conventions: `$PROJECT` = the FORGE clone on **shared storage** (not `$HOME`, see 3.1); `<account>`, `<partition>` = scheduler placeholders; `/path/to/refs/` = wherever your references live; `my_study.config`, `my_manifest.csv` = the two files you write. ## 1. What FORGE is and when to use it FORGE is a Nextflow pipeline for end-to-end analysis of single-cell and single-nucleus **multiome (RNA + ATAC)** data on HPC, from raw 10x (Cell Ranger ARC) or BD Rhapsody output. Two files — a manifest CSV and a short dataset config — drive ~108 processes in 13 workflow blocks: an **RNA arm** (CellBender → QC → scVI → scANVI/CellTypist → MAST DE → hdWGCNA), an **ATAC arm** (SnapATAC2 QC → peaks → scATAnno → Cicero CCANs → ChromVAR → SCENIC+/pycisTopic → scPRINTER footprinting), **multiome integration** (MOFA+ → MultiVI → MuData) and **communication/visualization** (CellChat → enhancer footprinting recipes A/B/C → genome browser tracks). RNA-only or ATAC-only runs are possible (`rna.run` / `atac.run`). | Stage | Tool | Citation | |---|---|---| | Ambient RNA correction | CellBender | Fleming et al. 2023, *Nat Methods* | | RNA QC, clustering | scanpy | Wolf et al. 2018, *Genome Biol* | | RNA integration | scVI | Lopez et al. 2018, *Nat Methods* | | RNA annotation | scANVI / CellTypist | Xu et al. 2021, *Mol Syst Biol*; Domínguez Conde et al. 2022, *Science* | | ATAC QC, peaks, clustering, DA | SnapATAC2 | Zhang et al. 2024, *Nat Methods* | | ATAC annotation | scATAnno | Jiang et al. 2025, *Genom Proteom Bioinform* | | Co-accessibility | Cicero | Pliner et al. 2018, *Mol Cell* | | TF motif enrichment | chromVAR | Schep et al. 2017, *Nat Methods* | | GRN / topic models | SCENIC+ / pycisTopic | Bravo González-Blas et al. 2023, *Nat Methods* | | TF footprinting | scPRINTER | Hu et al. 2025, *Nature* | | Co-expression networks | hdWGCNA | Morabito et al. 2023, *Cell Rep Methods* | | Cell–cell communication | CellChat | Jin et al. 2021, *Nat Commun* | | Multi-modal integration | MOFA+ / MultiVI | Argelaguet et al. 2020, *Genome Biol*; Ashuach et al. 2023, *Nat Methods* | | Differential expression | MAST | Finak et al. 2015, *Genome Biol* | Footprinting scores against JASPAR 2022; motif enrichment uses CIS-BP. **Validated genomes**: GRCh38/hg38 (Gencode v38), GRCm38/mm10 (Gencode vM10), GRCm39/mm39 (Gencode vM37) — not a whitelist. **Shipped examples**: `examples/nextflow_PBMC_Hs_10X.config` (public 10x *10k Human PBMCs, Multiome v1.0, Chromium X*) and `examples/nextflow_AD_Mm_10X.config` (Alzheimer's mouse, mm10; download link "to be provided"). BD Rhapsody configs in `configs/datasets/` "require collaborator metadata not yet released". ## 2. Requirements | Requirement | Value | Notes | |---|---|---| | Nextflow | **25.10.0** tested; supported `>=25.04.0, <26.0.0` | The README's "≥ 23.04" is stale: `nextflow.config` records `<= 24.10.5` fails to compile `main.nf`, `>= 26.04.6` fails to parse `nextflow.config`. | | Java | 17 | `java -version`; if missing, `module load java/17`. | | Singularity or Apptainer | ≥ 3.8 | On `$PATH` (`module load singularity` or `module load apptainer`). | | SLURM | any | The only shipped scheduler profile. | | GPU | **optional** | CPU-only end to end; see 2.1. | | High-memory node | ≥ 256 GB RAM | SCENIC+ and large reference atlases. | | Disk | ~600 GB references (full) + ~13–14 GB containers | A minimal RNA + ATAC run needs far less (§6). | ### 2.1 GPU tiers and the CPU-only reality | Tier | Stages | Notes | |---|---|---| | **A30-class** (≥ 24 GB VRAM) | `TRAIN_SCVI`, `TRAIN_SCANVI` | The only genuine A30 requirement — full-atlas training exhausts smaller cards. | | **V100-class** (16 GB VRAM) | `CELLBENDER`, `GPU_CHROMVAR`, `MULTIVI_*`, `MOFA_INTEGRATE` | Ample at these sizes. | | **CPU-only** | everything else (~100 of ~108 processes) | No GPU code path at all. | No stage *requires* a GPU: the tutorial completes all 94 tasks without one, running CellBender, MOFA+ and MultiVI on CPU (`scvi_accelerator = 'cpu'`). scVI/scANVI do **not** run in the tutorial — RNA annotation goes through CellTypist, so `TRAIN_SCVI` and `TRAIN_SCANVI` appear nowhere in its 25-process graph — and their CPU training was verified separately by the authors (`nextflow.config`'s `scvi_accelerator` note, 2026-08-06), not by the tutorial. **ChromVAR is the exception**: `bin/gpu_chromvar_nf.py` imports `cupy`/`rmm` at module scope with no CPU fallback (the tutorial sets `chromvar { run = false }`). For local runs `docs/setup/cluster.md` says to disable the GPU stages "rather than expecting a CPU fallback". Disabling ChromVAR has consequences worth knowing before you plan around it. `main.nf` gates the whole `ENHANCER_FOOTPRINTING_RECIPES` block on `enhancer_footprinting.run && cicero.run && chromvar.run && scprinter.run`, `differential_tf` on `chromvar.run`, and Cicero's ChromVAR-driven target selection (`cicero.use_chromvar_targets`, default `true`) on the same flag. Without a GPU you therefore cannot run TF motif enrichment, differential TF accessibility, or any of §8.5 step 3 (multi-scale footprinting, strips, TF–gene networks). Everything else — the RNA arm, ATAC QC/peaks/annotation, Cicero, MOFA+/MultiVI, CellChat, hdWGCNA, pycisTopic/SCENIC+ — runs on CPU. ### 2.2 The `NXF_VER` trap The `get.nextflow.io` installer always fetches the **newest** release, which is outside the window. An out-of-window Nextflow aborts while parsing the config with `Config parsing failed`, *before* FORGE's own version check can print anything friendlier. `export NXF_VER=25.10.0` **before** installing is the only real protection. ## 3. Install without sudo, then the 15-second pre-flight ### 3.1 Nextflow in a user directory, then clone `$HOME` on most clusters is small and quota-limited; a full run will not fit, so use a workspace on **shared storage visible from compute nodes** as `$PROJECT`. No `sudo` is needed; many clusters have no `nextflow` module. ```bash export NXF_VER=25.10.0 # BEFORE the install; add to ~/.bashrc too mkdir -p ~/bin && cd ~/bin curl -s https://get.nextflow.io | bash # writes ./nextflow here export PATH="$HOME/bin:$PATH" # add to ~/.bashrc to persist nextflow -version # should report 25.10.0 cd /path/to/your/workspace # NOT ~/bin git clone https://github.com/swaruplabUCI/FORGE.git cd FORGE && export PROJECT="$PWD" ``` ### 3.2 Prove the graph is sound (do this first, always) FORGE ships a fixture (`test_data/`, eleven placeholder files of a few hundred bytes each — the repository variously describes it as "7 KB" and "a few hundred KB"; measured, it is ~1.6 KB of content) and `configs/datasets/test_preview.config`, which enables every optional block. No containers, references, GPU or cluster needed: ```bash nextflow run main.nf -profile test -preview \ -c configs/datasets/test_preview.config ``` Expected, in about fifteen seconds: ```text WARN: Missing container files: [...]. Not required for -preview, but a real run needs them in singularity_cache/. PRE-FLIGHT CHECKLIST PASSED (8 checks): [OK] Manifest schema (2 rows) [OK] Species/genome consistency [OK] GTF files (5 paths validated) [OK] scATAnno reference atlas (stub_atlas.h5ad) [OK] MOFA mode (high_memory) [OK] scPRINTER genome (hg38) [OK] CellTypist model: Immune_All_Low.pkl [OK] Resource tier (test) Warnings: 1 (see above) ``` **The container warning is expected** on a fresh clone: a preview launches no task. For a real run the same check is an error (`Expected in singularity_cache/. Run container build/pull first.`); with containers built you see nine checks. **`-profile test`** strips every site-specific scheduler assumption via `configs/resource_tiers/test.config` (1 CPU / 1 GB, `clusterOptions = ''`), disables Singularity and redirects `outdir` to `results_test/` — it is for `-preview` only, **never for a real analysis**. `-preview` catches construction-time defects but nothing that only happens at task runtime. Print the merged configuration at any time with `nextflow config -profile cluster,singularity`. ## 4. The three core files ### 4.1 The manifest CSV (`params.metadata_file`) The **only** file that describes your data. One row per sample; `rna_file` and `fragment_file` are **filenames**, the directory goes in `data_dir`. ```csv sample_id,batch,sample_type,original_lane_id,rna_file,fragment_file,condition_group,data_dir my_sample,batch1,lane,L1,my_sample_raw_feature_bc_matrix.h5,my_sample_atac_fragments.tsv.gz,ConditionA,/data/my_study ``` | Column | Required | Meaning | |---|---|---| | `sample_id` | **yes** | Unique; joins RNA and ATAC; prefix on nearly every output. Blank rows silently skipped, duplicates a pre-flight error. | | `sample_type` | **yes** | Write exactly `lane`, lowercase. Vestigial (not a sequencing lane). The validator accepts `lane` or `demux` — `demux` is a BD demux path that additionally requires `fragment_file` — and its message reads `Expected 'lane' or 'demux' (case-sensitive)`. `main.nf`'s routing layer (`normalizeSampleType`) does lowercase and accept `rna`/`atac` as aliases, but the validator still rejects them: do not rely on the aliases. Unrelated strings pass validation and then match no channel, silently dropping the row. | | `batch` | yes in practice | Batch/group label (`docs/core/manifest.md`: "used for batch correction"); the lookup key into `params.batch_dirs` / `params.atac_batch_dirs` / `params.atac_coord_batch_dirs` when `data_dir` is absent, and carried in each sample's meta map. It is *not* the scVI batch axis — `TRAIN_SCVI` hard-codes `--batch_key sample`. | | `rna_file` | for RNA runs | 10x `*_raw_feature_bc_matrix.h5`; BD `*_RSEC_MolsPerCell_MEX.zip`; or a MEX directory (`matrix.mtx.gz`, `barcodes.tsv.gz`, `features.tsv.gz`). | | `fragment_file` | for ATAC runs | 10x `*_atac_fragments.tsv.gz`; BD `*_ATAC_Fragments.bed.gz`. No extension → `.bed.gz` appended. | | `condition_group` | for differential | e.g. `WT`/`TG`. **Single-condition datasets still need one label on every row.** Empty warns and defaults to `Control`. | | `data_dir` | see below | Absolute directory for the row's files; wins if present. Otherwise RNA rows resolve through `params.batch_dirs[batch]` and ATAC fragments through the separate `params.atac_batch_dirs[batch]`. | | `original_lane_id` | optional | Lane subdirectory, only for batches listed in `params.batch_dirs_use_lane_subdir`. | | `coord_data_dir` | ignored | Accepted by the manifest schema and documented in `docs/core/manifest.md`, but `resolveAtacCoordDir` has no caller anywhere in `main.nf` — the column has no effect. Leave it blank. | Per-batch alternative (relocatable; the shipped test and tutorial manifests use it) — set **both** maps, because RNA and ATAC resolve through different ones: `params.batch_dirs = [june: '/data/june_run', july: '/data/july_run']` and `params.atac_batch_dirs = [june: '/data/june_run', july: '/data/july_run']` (`atac_batch_dirs` is not declared in `nextflow.config`, so your config is the only place it comes from). A row with neither source stops with `No directory configured for batch` (RNA) or `No ATAC directory configured for batch 'june'. Set atac_batch_dirs.june in config.` (ATAC). Pre-flight checks required columns, `sample_id` uniqueness, `rna_file` existence, MEX completeness and config/manifest coherence; misspelled headers are named (`did you mean: sample_ID?`) but **never remapped**. `fragment_file` is *not* pre-flight checked; a bad one surfaces slightly later as `ATAC fragment file not found`. (`docs/core/manifest.md` is authoritative; `docs/setup/install.md` shows an older three-column example with full paths.) ### 4.2 The dataset config (`-c my_study.config`) `nextflow.config` holds every default (~1,000 lines) and is the only place a parameter is declared; you never edit it. Layers, later wins: `nextflow.config → resource tier → -profile → -c my_study.config → --flag`. `-c` **merges, it does not validate** — a misspelled key is silently ignored. Start small, expensive stages off: ```groovy params { species = 'human' // or 'mouse' — required, no default metadata_file = '/path/to/my_manifest.csv' outdir = 'results_my_study' resource_tier = 'small' // ALSO pass --resource_tier on the CLI (§8.2) // References (see §6) gtf_human_full = '/path/to/refs/gencode.v38.annotation.gtf' blacklist_bed = '/path/to/refs/hg38-blacklist.v2.bed.gz' // These MUST be set explicitly, even though they repeat gtf_human_full // above, but for two different reasons. scprinter.gtf_human/gtf_mouse are // interpolated from gtf_*_full when nextflow.config is parsed, BEFORE your // dataset config merges, so they keep the OLD value (usually the literal // string 'null'). cicero.gtf_full / gtf_plot are plain null defaults with // no fallback to gtf_human_full at all. Pre-flight rejects every one of // them ("resolves to 'null'"). cicero { gtf_full = '/path/to/refs/gencode.v38.annotation.gtf' gtf_plot = '/path/to/refs/gencode.v38.annotation.gtf' // pre-flight requires it once cicero.target_genes is non-empty outdir = "${params.outdir}/cicero" // re-declare AFTER outdir, as every shipped dataset config does } scprinter {