implement-and-scale-bioinformatics-pipeline

name

implement-and-scale-bioinformatics-pipeline

description

Implement a designed genomics workflow in the chosen engine, containerize and pin every tool for reproducibility, scale it with scatter/gather on HPC Slurm or cloud Batch (spot on the fault-tolerant steps), and validate it against a GIAB/GA4GH hap.py truth set — then produce a pipeline-validation report. Reach for this when the user asks "build this pipeline in Nextflow/Snakemake/WDL", "containerize and pin it for reproducibility", "scale/cost-optimize this on Slurm or cloud", or "benchmark our variant calls against GIAB". Used by `genomics-pipeline-engineer` (primary).

Skill: implement-and-scale-bioinformatics-pipeline

Invoked by: genomics-pipeline-engineer (primary). Also consulted by bioinformatics-workflow-architect to confirm the chosen engine/compute can actually run the workflow before finalizing the architecture.

When to invoke: "Build the pipeline in "; "containerize + pin it"; "scale / cost-optimize on Slurm or cloud"; "benchmark against GIAB/hap.py"; any move from a designed workflow to a running, reproducible, validated pipeline.

Output: the implemented workflow (per-step processes, containers, resource labels, params/config), the reproducibility manifest (pinned versions + lockfile + provenance), the scaling/cost plan, and a truth-set concordance report captured in the validation-report template.

Procedure

1. Start from the analysis plan. Take the step graph, reference build, accessory files, and truth set from design-genomics-analysis-workflow / ../../templates/analysis-plan-spec.md. Don't implement against a vague ask — an unspecified step graph produces an unmaintainable pipeline.
2. Implement one process per step. Each step (QC, trim, align, dedup, BQSR, call, joint-genotype / quantify) is a discrete process/rule with its own pinned container, a resource label (CPU/RAM/time), and typed inputs/outputs. Never a mega-script — the process graph is the unit of resume and reuse. Add a MultiQC roll-up at the end.
3. Containerize + pin for reproducibility. Per-process containers (Docker in dev → Apptainer/Singularity on HPC) or locked Conda/Bioconda envs; every tool at an exact version; commit the lockfile and a params file. Record the reference build + accessory-file checksums. Capture provenance (engine -with-trace/-with-report / Snakemake report, seeds).
4. Scale with scatter/gather + right-sized resources. Scatter per-sample and per genomic interval for alignment/calling; gather at joint-genotyping. Right-size resources from profiling a representative sample, not guesses. Use the engine's resume so a failure restarts from the last good step.
5. Cost-optimize the compute placement. On cloud, put the short, fault-tolerant, checkpointed steps on spot/preemptible with retries; keep long single-shot steps (a big joint-genotype gather) on-demand. Prefer CRAM over BAM, delete intermediates, keep data close to compute (watch egress). Record a before/after runtime + cost.
6. Validate against the truth set. For variant calling, run GA4GH hap.py against the GIAB truth VCF + confident-region BED and report precision / recall / F1 by variant type (SNV/indel) in the confident regions. For RNA/single-cell, check the expected QC signatures + spike-ins/markers. Regression-gate: a tool-version bump that degrades concordance must fail here, not in production.
7. Capture it in ../../templates/pipeline-validation-report.md — the engine/version, the pinned environment, the scaling/cost result, and the concordance numbers in one auditable page.

Worked example

User: "Build the WGS germline pipeline we designed in Nextflow, make it reproducible, run it on AWS Batch cheaply, and prove it works."

• Implement: DSL2 processes — fastqc → multiqc, fastp, bwamem2_align, markduplicates, bqsr, haplotypecaller (scattered over interval BED, gVCF), genomicsdb_import + genotypegvcfs (gather), vqsr, vep. Each with a pinned container and a resourceLabel.
• Reproducibility: each process pins its Biocontainer to an exact tag; a nextflow.config with a params file; -with-report -with-trace -with-timeline for provenance; reference (GRCh38 analysis set) + dbSNP checksums recorded; the container digests committed.
• Scale on AWS Batch: the AWS Batch executor; scatter HaplotypeCaller over ~can-be-many interval shards; the alignment shards + per-interval calling run on spot with errorStrategy 'retry' (reclaim = a cheap retry); the single-shot joint-genotype gather runs on-demand. CRAM output, intermediates cleaned. Before/after: record the spot-vs-on-demand cost delta.
• Validate: run the pipeline on GIAB HG002, then hap.py calls.vcf.gz against the HG002 truth VCF + confident BED → report SNV and indel precision/recall/F1 in the confident regions. Gate the pipeline on meeting the target before it's trusted for real samples.
• Report: all of it in the pipeline-validation report — engine version, pinned digests, scatter config, spot/cost result, and the hap.py concordance table.

Guardrails

• Implement against a captured analysis plan, never a vague ask.
• One process per step, one pinned container per process — mega-scripts kill resume, reuse, and reproducibility.
• Pin every version + commit a lockfile + record reference checksums — a floating solve or a latest tag is a non-reproducible result.
• Scatter/gather + right-size from profiling is the primary scaling lever, not bigger nodes.
• Spot only on fault-tolerant, retryable steps; keep long single-shot steps on-demand.
• Validate against GIAB/hap.py before claiming accuracy — report precision/recall/F1 by variant type in the confident regions, never "looks fine."
• Never mix reference builds/coordinates — the build + its matching accessory files are one set.
• Volatile facts (tool versions, GIAB truth-set versions, cloud pricing) carry a retrieval date; re-verify before shipping. See ../../knowledge/genomics-workflow-patterns-2026.md.