Authors reproducible Nextflow DSL2 pipelines built on reactive dataflow, where processes communicate only through channels and execution order is not guaranteed. Use when deciding channel/dataflow (Nextflow) vs rule-based (Snakemake) authoring; wiring queue vs value channels and fixing shared-reference exhaustion with .first(); composing DSL2 modules and subworkflows with take/main/emit; selecting container/conda profiles and pinning images by digest for portability across local/SLURM/LSF/AWS Batch/Google Batch/Kubernetes executors; diagnosing why -resume misses the cache (nondeterministic input order, mtime on network filesystems, mutable :latest tags) with cache 'lenient' and -dump-hashes; managing work/ vs publishDir and dynamic retry escalation; and choosing whether to adopt an nf-core community pipeline or author from scratch.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
A direct command skips the review prompt. Inspect the source before running it.
Authors reproducible Nextflow DSL2 pipelines built on reactive dataflow, where processes communicate only through channels and execution order is not guaranteed. Use when deciding channel/dataflow (Nextflow) vs rule-based (Snakemake) authoring; wiring queue vs value channels and fixing shared-reference exhaustion with .first(); composing DSL2 modules and subworkflows with take/main/emit; selecting container/conda profiles and pinning images by digest for portability across local/SLURM/LSF/AWS Batch/Google Batch/Kubernetes executors; diagnosing why -resume misses the cache (nondeterministic input order, mtime on network filesystems, mutable :latest tags) with cache 'lenient' and -dump-hashes; managing work/ vs publishDir and dynamic retry escalation; and choosing whether to adopt an nf-core community pipeline or author from scratch.
Before using code patterns, verify installed versions match. If versions differ:
CLI: <tool> --version then <tool> --help to confirm flags
If code throws ImportError, AttributeError, or TypeError, introspect the installed
package and adapt the example to match the actual API rather than retrying.
Note: Nextflow is DSL2-only (DSL1 was removed in 22.12) and calendar-versioned (24.x/25.x), so any single-script DSL1 tutorial is dead. The nf-validation plugin is deprecated in favor of . A strict, statically-analyzable syntax (VS Code language server) is opt-in now and default in a later release; writing to it future-proofs a pipeline. Pin container images by immutable digest (), never a moving tag such as , or both reproducibility and break.
nf-schema
@sha256:
:latest
-resume
Nextflow Pipelines
"Build a scalable, reproducible pipeline with Nextflow" -> Wire containerized processes together with asynchronous channels (reactive dataflow), so the engine fires each task as soon as its inputs are ready, caches completed tasks for -resume, and moves unchanged across executors by swapping a profile.
CLI: nextflow run main.nf -profile docker -resume
Groovy: DSL2 process / workflow / channel-operator syntax
The governing principle: Nextflow is reactive dataflow - processes talk ONLY through channels, execution order is NOT guaranteed
Nextflow is push-model dataflow: processes are pure functions wired together by asynchronous channels, every channel item is a future, and a process fires a task the instant a complete set of inputs is available on ALL its input channels. There is no target file, no backward DAG, no filename-to-rule matching. This is the opposite of Snakemake/CWL/WDL, which are pull/goal-oriented (name a target output, the engine walks a dependency DAG backward to decide what runs). Almost every downstream trap traces back to this one axis:
The model ENABLES truly dynamic pipelines: a process emits N files computed at runtime, the next process fans out over all N, with no DAG known in advance. Pull engines need a checkpoint/scatter escape hatch for this.
The COST is that "what will run" is not knowable by reading the script top-to-bottom. Independent branches interleave nondeterministically, so SAMPLE_A may finish after SAMPLE_Z. Never write logic that assumes order; carry an explicit sample key through the tuple instead.
"Why did only the first sample run?" and "why did -resume re-run everything?" are the two most common support questions, and both are direct symptoms of the dataflow model (see queue-vs-value channels and cache-miss diagnosis below).
A second principle sits above the engine: the DAG buys reproducibility of workflow LOGIC and nothing else automatically (Wratten et al. 2021 Nat Methods 18:1161-1168). A clean pipeline over unpinned tools is NOT reproducible. Pin the software environment (container by digest, conda by lockfile), the reference data and params, and control thread/locale/arch leaks (Grüning et al. 2018 Cell Syst 6:631-635). The engine gives one layer; the author pins the rest.
Decision: Nextflow vs the other engines
Axis
Nextflow (DSL2)
Snakemake
WDL (Cromwell/miniwdl)
CWL
Model
reactive dataflow (push, no target)
pull/goal (target -> backward DAG)
pull/goal (declared outputs)
pull/goal (typed, declared)
Dynamic DAG (shape depends on runtime data)
native, trivial
checkpoints (bolted on)
scatter (static-ish)
limited
Cloud/executor portability
best-in-class (swap by profile)
good (v8 plugins, catching up)
strong on GCP/Terra
via Toil/Arvados
Community pipelines
nf-core (largest, curated)
Workflow Catalog (smaller)
WARP (Broad)
limited
Best when
cloud/production, dynamic pipelines, want nf-core
Python shop, HPC, file-pattern logic
Terra/AnVIL, GATK best practices
vendor-neutral portability, regulated
Honest take: Nextflow wins on executor portability, nf-core, and dynamic pipelines; Snakemake wins on approachability for Python users. Pick by the ecosystem to integrate with, not by benchmarks.
Decision: queue vs value channel (THE #1 footgun)
Need
Channel type
Create with
Exhaustion behavior
One item consumed by one task (per-sample reads)
queue
Channel.of, .fromPath, .fromFilePairs, .splitCsv
consumed once, then empty forever
A shared value reused on EVERY task (a reference/index)
value (singleton)
Channel.value(x), .first(), .collect(), or a bare param
read unlimited times, never exhausted
The firing rule to memorize: a process launches a new task only when EVERY input channel can supply an item; when a queue input drains, no more tasks fire even if other inputs still have items. So a shared reference passed as a queue channel is consumed by the first sample and every later sample silently never runs (exit 0, no error). Corollary: if all of a process's inputs are value channels its outputs are value channels too; if any input is a queue channel the outputs are queue channels.
Decision: operator selection (and the silent-data-loss traps)
Operator
Does
Trap / when-wrong
map
transform each item
pure only; no I/O side effects
collect
ALL items -> one list item (queue -> value)
blocks until upstream closes; gathers inputs for one aggregating task (MultiQC)
groupTuple
group by key into [key, [items]]
bare form WAITS for the whole channel to close (serialization/deadlock); pass size: N or groupKey(key, n); SORT the grouped list or resume breaks
join
inner-join two channels by key
SILENTLY DROPS non-matching keys by default; use remainder: true or failOnMismatch: true
combine
Cartesian product (optional by:)
intentional all-vs-all; distinct from join (1:1 merge)
mix
interleave channels into one
order not preserved; pool outputs before a collect
branch
route items to named sub-channels
the DSL2 idiom for conditional routing (single_end vs paired)
first
first item as a VALUE channel
THE queue -> value fix for shared references
ifEmpty
supply a default if empty
guards the "empty branch silently vanishes" trap
Decision: executor (the portability payoff)
Target
executor
Best when
Watch
Laptop/dev
local
development, tiny data, -stub wiring tests
one machine only
On-prem HPC
slurm, lsf, sge, pbs
shared cluster, on-prem data
tune queueSize/submitRateLimit; scratch true on slow shared FS
Cloud batch
awsbatch, google-batch, azurebatch
elastic scale, no on-prem HPC
input localization copy dominates cost/time; Wave + Fusion cut it
Kubernetes
k8s
already running K8s
more setup overhead
Never bake the executor into pipeline code; always set it in a profile so the same code moves across all of them.
Process and DSL2 modules (take / main / emit)
A DSL2 module wraps one tool as a process and can be included and called multiple times (aliased), which DSL1 could not. Subworkflows compose modules with named inputs/outputs.
// modules/fastqc.nf -- one tool, reusable, tested in isolation
process FASTQC {
tag "${meta.id}" // meta map threads sample identity through every operator
container 'quay.io/biocontainers/fastqc:0.12.1--hdfd78af_0' // pin an immutable tag/digest, never :latest
label 'process_low' // maps to central resource config, decoupled from module code
input:
tuple val(meta), path(reads) // nf-core convention: [ meta, files ], meta = [id:'x', single_end:false]
output:
tuple val(meta), path('*.zip'), emit: zip // meta round-trips so downstream always knows the sample
script:
"""
fastqc -t ${task.cpus} ${reads}
"""
}
// subworkflows/qc.nf -- take/main/emit names the interface
include { FASTQC } from '../modules/fastqc'
include { MULTIQC } from '../modules/multiqc'
workflow QC {
take:
reads
main:
FASTQC(reads)
MULTIQC(FASTQC.out.zip.collect()) // collect() gathers all samples' zips into ONE aggregating task
emit:
report = MULTIQC.out.report // access as QC.out.report from the caller
}
Channels: the shared-reference fix, explained
workflow {
reads_ch = Channel.fromFilePairs(params.reads) // queue: [id, [r1, r2]] per sample -- consumed once each
index_ch = Channel.fromPath(params.index) // queue: ONE item, the shared index
// BUG if written ALIGN(reads_ch, index_ch): the index is consumed by sample 1,
// its queue is then empty, and samples 2..N silently never fire (exit 0, no error).
// .first() converts the queue to a VALUE channel, reusable on every task invocation.
ALIGN(reads_ch, index_ch.first())
}
Resume: the task hash, and diagnosing a cache miss
-resume reuses a task only on an EXACT hit of the task hash, computed from the input file identities, the resolved script text, the container reference, and input values/params. A single-bit change in any component busts the cache and re-runs the task. The notorious silent causes:
Nondeterministic input ordering (collect/groupTuple/glob expansion) -> the ordered list is part of the hash. Fix: toSortedList() or .map{ k, v -> [k, v.sort()] }.
Mutable container tags (:latest, or a re-pushed version) -> pin by digest.
mtime hashing on network filesystems (Lustre/NFS) -> -resume works locally but misses on the cluster. Fix: cache 'lenient' (hashes size + path, ignores mtime).
Absolute paths, dates, $RANDOM, or hostname baked into the script string -> the script hash changes every run.
A deleted or moved work/ -> the hash hits the DB but the task dir is gone, forcing a re-run.
process ALIGN {
// 'lenient' skips mtime -- the single most useful resume fix on HPC/cloud shared filesystems.
// 'deep' hashes full file CONTENT (slower, robust when metadata lies); 'false' never caches.
cache 'lenient'
// ...
}
Definitive diagnosis: run both executions with -dump-hashes and diff which hash component differed, or nextflow log <run_name> -f hash,name,status,workdir to compare per-task hashes across runs. Everything else is guessing.
work/ is truth; publishDir is a side effect
Every task runs in an isolated work/<hash>/ dir holding the real outputs plus the forensic trail (.command.sh resolved script, .command.log, .exitcode). That directory IS the pipeline's output store and the ONLY thing -resume reads. publishDir merely copies or symlinks SELECTED outputs to a human-friendly location, and its failure can be SILENT because the task itself exited 0 in work/. Consequences:
mode: 'symlink' (default) breaks if work/ is later deleted; mode: 'copy' is safe to delete afterward; mode: 'move' breaks -resume (the output leaves work/), so use it only for terminal outputs.
Never rm -rf work/ if a resume might be wanted; use nextflow clean (which prunes the cache DB consistently). "Outputs missing but the pipeline succeeded" almost always means looking in publishDir instead of work/<hash>/.
Resources: dynamic retry escalation
process BIG {
// 137=SIGKILL/OOM, 143=SIGTERM (SLURM wall-time kill); the 130..145 signal band + 104 (transient I/O) retry, fail fast otherwise.
errorStrategy { task.exitStatus in ((130..145) + 104) ? 'retry' : 'terminate' }
maxRetries 3
memory { 8.GB * task.attempt } // task.attempt is 1-based; escalates 8 -> 16 -> 24 -> 32 GB
time { 4.h * task.attempt } // a transient OOM auto-escalates instead of killing the run
script:
"""
memory_intensive_command
"""
}
errorStrategy values are 'terminate' (default), 'retry', 'ignore' (drop the failed task's outputs and continue over survivors), and 'finish' (graceful drain). The nf-core process.resourceLimits directive (Nextflow 24.04+, which replaced the pre-3.0 check_max pattern) clamps the escalated request to the machine/queue ceiling so 8.GB * task.attempt never asks for more than a node has.
Run with -profile slurm,singularity (comma-separated, NO spaces; later profiles override earlier).
Adopt an nf-core pipeline before authoring
For any mainstream analysis (RNA-seq, variant calling, ATAC, methylation, amplicon), a curated nf-core/<pipeline> already encodes years of QC, containerized modules, nf-test regression tests, and institutional configs. Reinventing it is months of work and worse QC. Pin the revision: nextflow run nf-core/rnaseq -r 3.14.0 -profile test,docker --outdir results. DIY is justified only for genuinely novel logic. See workflow-management/nf-core-pipelines for running, configuring, and building samplesheets against community pipelines; this skill covers AUTHORING.
Common Errors
Symptom
Cause
Fix
Only the first sample processed, exit 0, no error
shared reference on a queue channel, exhausted after task 1
.first() / Channel.value on the reference
Pipeline hangs at a grouping step
groupTuple with no size on a channel that never closes
size: N or groupKey(key, n)
Some samples silently disappear mid-pipeline
join dropped non-matching keys
remainder: true or failOnMismatch: true
-resume re-runs everything
nondeterministic input order, or :latest tag, or mtime on network FS
workflow-management/nf-core-pipelines - Run and configure community pipelines (the RUNNING counterpart to this authoring skill)
workflow-management/snakemake-workflows - Pull/goal-oriented alternative for Python shops and file-pattern logic
workflows/rnaseq-to-de - End-to-end RNA-seq quantification to differential expression
read-qc/quality-reports - QC steps a pipeline orchestrates (FastQC/MultiQC)
References
Di Tommaso P, Chatzou M, Floden EW, Prieto Barja P, Palumbo E, Notredame C. 2017. Nextflow enables reproducible computational workflows. Nat Biotechnol 35(4):316-319.
Ewels PA, Peltzer A, Fillinger S, Patel H, Alneberg J, Wilm A, Garcia MU, Di Tommaso P, Nahnsen S. 2020. The nf-core framework for community-curated bioinformatics pipelines. Nat Biotechnol 38(3):276-278.
Wratten L, Wilm A, Göke J. 2021. Reproducible, scalable, and shareable analysis pipelines with bioinformatics workflow managers. Nat Methods 18:1161-1168.
Grüning B, Chilton J, Köster J, et al. 2018. Practical computational reproducibility in the life sciences. Cell Syst 6(6):631-635.
