bio-workflow-management-nextflow-pipelines

name	bio-workflow-management-nextflow-pipelines
description	Authors reproducible Nextflow DSL2 pipelines built on reactive dataflow, where processes communicate only through channels and execution order is not guaranteed. Use when deciding channel/dataflow (Nextflow) vs rule-based (Snakemake) authoring; wiring queue vs value channels and fixing shared-reference exhaustion with .first(); composing DSL2 modules and subworkflows with take/main/emit; selecting container/conda profiles and pinning images by digest for portability across local/SLURM/LSF/AWS Batch/Google Batch/Kubernetes executors; diagnosing why -resume misses the cache (nondeterministic input order, mtime on network filesystems, mutable :latest tags) with cache 'lenient' and -dump-hashes; managing work/ vs publishDir and dynamic retry escalation; and choosing whether to adopt an nf-core community pipeline or author from scratch.
tool_type	cli
primary_tool	Nextflow

Version Compatibility

Reference examples tested with: Nextflow 24.04+, fastp 0.23+, Salmon 1.10+, MultiQC 1.21+

Before using code patterns, verify installed versions match. If versions differ:

• CLI: <tool> --version then <tool> --help to confirm flags

If code throws ImportError, AttributeError, or TypeError, introspect the installed package and adapt the example to match the actual API rather than retrying.

Note: Nextflow is DSL2-only (DSL1 was removed in 22.12) and calendar-versioned (24.x/25.x), so any single-script DSL1 tutorial is dead. The nf-validation plugin is deprecated in favor of nf-schema. A strict, statically-analyzable syntax (VS Code language server) is opt-in now and default in a later release; writing to it future-proofs a pipeline. Pin container images by immutable digest (@sha256:), never a moving tag such as :latest, or both reproducibility and -resume break.

Nextflow Pipelines

"Build a scalable, reproducible pipeline with Nextflow" -> Wire containerized processes together with asynchronous channels (reactive dataflow), so the engine fires each task as soon as its inputs are ready, caches completed tasks for -resume, and moves unchanged across executors by swapping a profile.

• CLI: nextflow run main.nf -profile docker -resume
• Groovy: DSL2 process / workflow / channel-operator syntax

The governing principle: Nextflow is reactive dataflow - processes talk ONLY through channels, execution order is NOT guaranteed

Nextflow is push-model dataflow: processes are pure functions wired together by asynchronous channels, every channel item is a future, and a process fires a task the instant a complete set of inputs is available on ALL its input channels. There is no target file, no backward DAG, no filename-to-rule matching. This is the opposite of Snakemake/CWL/WDL, which are pull/goal-oriented (name a target output, the engine walks a dependency DAG backward to decide what runs). Almost every downstream trap traces back to this one axis:

• The model ENABLES truly dynamic pipelines: a process emits N files computed at runtime, the next process fans out over all N, with no DAG known in advance. Pull engines need a checkpoint/scatter escape hatch for this.
• The COST is that "what will run" is not knowable by reading the script top-to-bottom. Independent branches interleave nondeterministically, so SAMPLE_A may finish after SAMPLE_Z. Never write logic that assumes order; carry an explicit sample key through the tuple instead.
• "Why did only the first sample run?" and "why did -resume re-run everything?" are the two most common support questions, and both are direct symptoms of the dataflow model (see queue-vs-value channels and cache-miss diagnosis below).

A second principle sits above the engine: the DAG buys reproducibility of workflow LOGIC and nothing else automatically (Wratten et al. 2021 Nat Methods 18:1161-1168). A clean pipeline over unpinned tools is NOT reproducible. Pin the software environment (container by digest, conda by lockfile), the reference data and params, and control thread/locale/arch leaks (Grüning et al. 2018 Cell Syst 6:631-635). The engine gives one layer; the author pins the rest.

Decision: Nextflow vs the other engines

Axis	Nextflow (DSL2)	Snakemake	WDL (Cromwell/miniwdl)	CWL
Model	reactive dataflow (push, no target)	pull/goal (target -> backward DAG)	pull/goal (declared outputs)	pull/goal (typed, declared)
Dynamic DAG (shape depends on runtime data)	native, trivial	checkpoints (bolted on)	scatter (static-ish)	limited
Cloud/executor portability	best-in-class (swap by profile)	good (v8 plugins, catching up)	strong on GCP/Terra	via Toil/Arvados
Community pipelines	nf-core (largest, curated)	Workflow Catalog (smaller)	WARP (Broad)	limited
Best when	cloud/production, dynamic pipelines, want nf-core	Python shop, HPC, file-pattern logic	Terra/AnVIL, GATK best practices	vendor-neutral portability, regulated

Honest take: Nextflow wins on executor portability, nf-core, and dynamic pipelines; Snakemake wins on approachability for Python users. Pick by the ecosystem to integrate with, not by benchmarks.

Decision: queue vs value channel (THE #1 footgun)

Need	Channel type	Create with	Exhaustion behavior
One item consumed by one task (per-sample reads)	queue	Channel.of, .fromPath, .fromFilePairs, .splitCsv	consumed once, then empty forever
A shared value reused on EVERY task (a reference/index)	value (singleton)	Channel.value(x), .first(), .collect(), or a bare param	read unlimited times, never exhausted

The firing rule to memorize: a process launches a new task only when EVERY input channel can supply an item; when a queue input drains, no more tasks fire even if other inputs still have items. So a shared reference passed as a queue channel is consumed by the first sample and every later sample silently never runs (exit 0, no error). Corollary: if all of a process's inputs are value channels its outputs are value channels too; if any input is a queue channel the outputs are queue channels.

Decision: operator selection (and the silent-data-loss traps)

Operator	Does	Trap / when-wrong
map	transform each item	pure only; no I/O side effects
collect	ALL items -> one list item (queue -> value)	blocks until upstream closes; gathers inputs for one aggregating task (MultiQC)
groupTuple	group by key into [key, [items]]	bare form WAITS for the whole channel to close (serialization/deadlock); pass size: N or groupKey(key, n); SORT the grouped list or resume breaks
join	inner-join two channels by key	SILENTLY DROPS non-matching keys by default; use remainder: true or failOnMismatch: true
combine	Cartesian product (optional by:)	intentional all-vs-all; distinct from join (1:1 merge)
mix	interleave channels into one	order not preserved; pool outputs before a collect
branch	route items to named sub-channels	the DSL2 idiom for conditional routing (single_end vs paired)
first	first item as a VALUE channel	THE queue -> value fix for shared references
ifEmpty	supply a default if empty	guards the "empty branch silently vanishes" trap

Decision: executor (the portability payoff)

Target	executor	Best when	Watch
Laptop/dev	local	development, tiny data, -stub wiring tests	one machine only
On-prem HPC	slurm, lsf, sge, pbs	shared cluster, on-prem data	tune queueSize/submitRateLimit; scratch true on slow shared FS
Cloud batch	awsbatch, google-batch, azurebatch	elastic scale, no on-prem HPC	input localization copy dominates cost/time; Wave + Fusion cut it
Kubernetes	k8s	already running K8s	more setup overhead

Never bake the executor into pipeline code; always set it in a profile so the same code moves across all of them.

Process and DSL2 modules (take / main / emit)

A DSL2 module wraps one tool as a process and can be included and called multiple times (aliased), which DSL1 could not. Subworkflows compose modules with named inputs/outputs.

// modules/fastqc.nf -- one tool, reusable, tested in isolation
process FASTQC {
    tag "${meta.id}"                                   // meta map threads sample identity through every operator
    container 'quay.io/biocontainers/fastqc:0.12.1--hdfd78af_0'  // pin an immutable tag/digest, never :latest
    label 'process_low'                                // maps to central resource config, decoupled from module code

    input:
    tuple val(meta), path(reads)                       // nf-core convention: [ meta, files ], meta = [id:'x', single_end:false]

    output:
    tuple val(meta), path('*.zip'), emit: zip          // meta round-trips so downstream always knows the sample

    script:
    """
    fastqc -t ${task.cpus} ${reads}
    """
}

// subworkflows/qc.nf -- take/main/emit names the interface
include { FASTQC } from '../modules/fastqc'
include { MULTIQC } from '../modules/multiqc'

workflow QC {
    take:
    reads

    main:
    FASTQC(reads)
    MULTIQC(FASTQC.out.zip.collect())                  // collect() gathers all samples' zips into ONE aggregating task

    emit:
    report = MULTIQC.out.report                        // access as QC.out.report from the caller
}

Channels: the shared-reference fix, explained

workflow {
    reads_ch = Channel.fromFilePairs(params.reads)     // queue: [id, [r1, r2]] per sample -- consumed once each
    index_ch = Channel.fromPath(params.index)          // queue: ONE item, the shared index

    // BUG if written ALIGN(reads_ch, index_ch): the index is consumed by sample 1,
    // its queue is then empty, and samples 2..N silently never fire (exit 0, no error).
    // .first() converts the queue to a VALUE channel, reusable on every task invocation.
    ALIGN(reads_ch, index_ch.first())
}

Resume: the task hash, and diagnosing a cache miss

-resume reuses a task only on an EXACT hit of the task hash, computed from the input file identities, the resolved script text, the container reference, and input values/params. A single-bit change in any component busts the cache and re-runs the task. The notorious silent causes:

• Nondeterministic input ordering (collect/groupTuple/glob expansion) -> the ordered list is part of the hash. Fix: toSortedList() or .map{ k, v -> [k, v.sort()] }.
• Mutable container tags (:latest, or a re-pushed version) -> pin by digest.
• mtime hashing on network filesystems (Lustre/NFS) -> -resume works locally but misses on the cluster. Fix: cache 'lenient' (hashes size + path, ignores mtime).
• Absolute paths, dates, $RANDOM, or hostname baked into the script string -> the script hash changes every run.
• A deleted or moved work/ -> the hash hits the DB but the task dir is gone, forcing a re-run.

process ALIGN {
    // 'lenient' skips mtime -- the single most useful resume fix on HPC/cloud shared filesystems.
    // 'deep' hashes full file CONTENT (slower, robust when metadata lies); 'false' never caches.
    cache 'lenient'
    // ...
}

Definitive diagnosis: run both executions with -dump-hashes and diff which hash component differed, or nextflow log <run_name> -f hash,name,status,workdir to compare per-task hashes across runs. Everything else is guessing.

work/ is truth; publishDir is a side effect

Every task runs in an isolated work/<hash>/ dir holding the real outputs plus the forensic trail (.command.sh resolved script, .command.log, .exitcode). That directory IS the pipeline's output store and the ONLY thing -resume reads. publishDir merely copies or symlinks SELECTED outputs to a human-friendly location, and its failure can be SILENT because the task itself exited 0 in work/. Consequences:

• mode: 'symlink' (default) breaks if work/ is later deleted; mode: 'copy' is safe to delete afterward; mode: 'move' breaks -resume (the output leaves work/), so use it only for terminal outputs.
• Never rm -rf work/ if a resume might be wanted; use nextflow clean (which prunes the cache DB consistently). "Outputs missing but the pipeline succeeded" almost always means looking in publishDir instead of work/<hash>/.

Resources: dynamic retry escalation

process BIG {
    // 137=SIGKILL/OOM, 143=SIGTERM (SLURM wall-time kill); the 130..145 signal band + 104 (transient I/O) retry, fail fast otherwise.
    errorStrategy { task.exitStatus in ((130..145) + 104) ? 'retry' : 'terminate' }
    maxRetries 3
    memory { 8.GB * task.attempt }                     // task.attempt is 1-based; escalates 8 -> 16 -> 24 -> 32 GB
    time   { 4.h  * task.attempt }                     // a transient OOM auto-escalates instead of killing the run

    script:
    """
    memory_intensive_command
    """
}

errorStrategy values are 'terminate' (default), 'retry', 'ignore' (drop the failed task's outputs and continue over survivors), and 'finish' (graceful drain). The nf-core process.resourceLimits directive (Nextflow 24.04+, which replaced the pre-3.0 check_max pattern) clamps the escalated request to the machine/queue ceiling so 8.GB * task.attempt never asks for more than a node has.

Executors and profiles: portability

// nextflow.config -- executor lives in a profile, never in the pipeline code
profiles {
    docker      { docker.enabled = true }
    singularity { singularity.enabled = true }
    slurm {
        process.executor = 'slurm'
        executor { queueSize = 100; submitRateLimit = '10/1min' }   // avoid hammering the scheduler
    }
    awsbatch {
        process.executor = 'awsbatch'
        aws.region = 'us-east-1'
    }
}

process {
    cpus = 2; memory = '4 GB'; time = '1h'             // sane defaults
    withLabel: 'process_high' { cpus = 16; memory = '64 GB'; time = '12h' }  // labels centralize per-tier tuning
}

Run with -profile slurm,singularity (comma-separated, NO spaces; later profiles override earlier).

Adopt an nf-core pipeline before authoring

For any mainstream analysis (RNA-seq, variant calling, ATAC, methylation, amplicon), a curated nf-core/<pipeline> already encodes years of QC, containerized modules, nf-test regression tests, and institutional configs. Reinventing it is months of work and worse QC. Pin the revision: nextflow run nf-core/rnaseq -r 3.14.0 -profile test,docker --outdir results. DIY is justified only for genuinely novel logic. See workflow-management/nf-core-pipelines for running, configuring, and building samplesheets against community pipelines; this skill covers AUTHORING.

Common Errors

Symptom	Cause	Fix
Only the first sample processed, exit 0, no error	shared reference on a queue channel, exhausted after task 1	.first() / Channel.value on the reference
Pipeline hangs at a grouping step	groupTuple with no size on a channel that never closes	size: N or groupKey(key, n)
Some samples silently disappear mid-pipeline	join dropped non-matching keys	remainder: true or failOnMismatch: true
-resume re-runs everything	nondeterministic input order, or :latest tag, or mtime on network FS	sort inputs; pin container digest; cache 'lenient'
Resume works locally, misses on the cluster	mtime unreliable on Lustre/NFS	cache 'lenient'
Outputs missing but the pipeline "succeeded"	publishDir failed silently, or looked in publishDir not work/	check work/<hash>/; use mode: 'copy'
Resume broken after cleanup	deleted work/	never rm -rf work/; use nextflow clean
OOM kills a long run near the end	fixed memory, no escalation	memory { 8.GB * task.attempt } + conditional retry
Wrong result, no error, after a base image update	mutable tag served a stale cache hit	pin by digest; cache 'deep' for critical inputs
Huge cloud bill / slow S3 pipeline	explicit stage-in/out copies of large files	Wave + Fusion (POSIX over object store)
-profile test docker ignores docker	space instead of comma	-profile test,docker

Related Skills

• workflow-management/nf-core-pipelines - Run and configure community pipelines (the RUNNING counterpart to this authoring skill)
• workflow-management/snakemake-workflows - Pull/goal-oriented alternative for Python shops and file-pattern logic
• workflows/rnaseq-to-de - End-to-end RNA-seq quantification to differential expression
• read-qc/quality-reports - QC steps a pipeline orchestrates (FastQC/MultiQC)

References

• Di Tommaso P, Chatzou M, Floden EW, Prieto Barja P, Palumbo E, Notredame C. 2017. Nextflow enables reproducible computational workflows. Nat Biotechnol 35(4):316-319.
• Ewels PA, Peltzer A, Fillinger S, Patel H, Alneberg J, Wilm A, Garcia MU, Di Tommaso P, Nahnsen S. 2020. The nf-core framework for community-curated bioinformatics pipelines. Nat Biotechnol 38(3):276-278.
• Wratten L, Wilm A, Göke J. 2021. Reproducible, scalable, and shareable analysis pipelines with bioinformatics workflow managers. Nat Methods 18:1161-1168.
• Grüning B, Chilton J, Köster J, et al. 2018. Practical computational reproducibility in the life sciences. Cell Syst 6(6):631-635.