write-nextflow-local-modules

name

write-nextflow-local-modules

description

How to create or update local Nextflow modules for any pipeline. Use this skill whenever the user asks to create, write, scaffold, add, or update local Nextflow modules — even if they just say "add a module for X" or "write a module that does Y". Local modules live at modules/local/<name>/ and resemble nf-core modules but with relaxed conventions: no nf-core lint required (only nextflow lint), local test fixtures instead of a separate test-datasets repo, nf-test process tests by default, and the modern topic-channel version output instead of versions.yml.

Writing Local Nextflow Modules

If following nf-core standards, local modules live alongside nf-core modules in modules/local/<module_name>/ and follow the same structural conventions, but without nf-core linting requirements (nextflow lint only).

Directory scaffold

bin/
└── <script_name>.py     # executable custom logic called by module script blocks
modules/local/<module_name>/
├── main.nf           # process definition
├── meta.yml          # module documentation
├── environment.yml   # conda dependencies
└── tests/
    ├── main.nf.test          # nf-test file
    ├── main.nf.test.snap     # snapshot (generated by running nf-test)
    └── fixtures/             # minimal local test data
        └── <test_input>.<ext>

Create bin/ whenever the module needs custom Bash, Python, R, or other scripting logic beyond a simple one-line command. Keep substantial custom logic out of modules/local/<module_name>/main.nf by default; the module should expose the Nextflow interface and call a script from bin/.

Create tests/main.nf.test, tests/main.nf.test.snap, and minimal tests/fixtures/ data as part of normal local-module work. Do not skip nf-tests just because the current repository has no examples; use this skill's template and run nf-test to generate the snapshot. Omit tests only when the user explicitly scopes the task to process code/config without tests, and say that tests were intentionally omitted.

---

main.nf

Process name in UPPER_SNAKE_CASE matching the module directory name.

process MODULE_NAME {
    tag "$meta.id"
    label 'process_single'   // choose: process_single, process_low, process_medium, process_high

    conda "${moduleDir}/environment.yml"
    container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
        '<singularity_sha256_blob_url>' :
        '<docker_image_url>' }"

    input:
    tuple val(meta), path(input_file)   // prefer one tuple per logical group
    val(threshold)                       // plain vals don't need a meta wrapper

    output:
    tuple val(meta), path("${prefix}.ext"), emit: result
    // one emit per tool/library — no versions.yml heredoc needed
    tuple val("${task.process}"), val('tool_name'), eval("tool --version 2>&1 | sed 's/tool //'"), emit: versions_tool, topic: versions

    when:
    task.ext.when == null || task.ext.when

    script:
    def args   = task.ext.args   ?: ''
    def prefix = task.ext.prefix ?: "${meta.id}"
    """
    script_name.py \\
        --input  ${input_file} \\
        --output ${prefix}.ext \\
        ${args}
    """

    stub:
    def prefix = task.ext.prefix ?: "${meta.id}"
    """
    touch ${prefix}.ext
    """
}

Input conventions

• Always include val(meta) as the first element of every file-containing tuple.
• Prefer a single tuple over multiple tuples when inputs logically belong together: tuple val(meta), path(file1), path(file2) rather than two separate tuples.
• Use separate tuples when inputs have independent identities (different samples, different meta maps).
• Plain scalars (val) don't need a meta wrapper — just val(threshold).

Version output — topic channel, not versions.yml

Use the topic: versions convention. One emit line per tool or library:

// CLI tool
tuple val("${task.process}"), val('mytool'), eval("mytool --version 2>&1 | sed 's/mytool //'"), emit: versions_mytool, topic: versions

// Python package
tuple val("${task.process}"), val('biopython'), eval("python -c \"import importlib.metadata; print(importlib.metadata.version('biopython'))\""), emit: versions_biopython, topic: versions

// Python itself
tuple val("${task.process}"), val('python'), eval("python --version 2>&1 | sed 's/Python //'"), emit: versions_python, topic: versions

The sed pattern must reduce output to a bare version string (e.g. 1.85, 3.13.1).

Stub block

Just create empty outputs. No heredoc for versions — topic-channel emits work the same in stub:

stub:
def prefix = task.ext.prefix ?: "${meta.id}"
"""
touch ${prefix}.ext
"""

For directories: mkdir -p dir && touch dir/placeholder For compressed files: echo "" | gzip > ${prefix}.gz

Script section: default to bin/ scripts

Use the module script: block as a thin command wrapper. Simple one-line Bash commands, shell builtins, pipelines, or direct tool invocations may stay inline. If the logic is custom Bash, Python, R, or more than a trivial one-line CLI invocation, put that logic in an executable script under the pipeline's top-level bin/ directory and call it from the module.

Inline script bodies are acceptable only when:

• the command is a simple one-line Bash/tool invocation such as mkdir -p out or samtools sort ..., or
• the user explicitly asks to keep the full Bash/Python body inside the module for teaching or prototyping.

Do not write Python heredocs or multi-line Bash programs directly inside local module script: blocks by default. Nextflow adds bin/ to PATH, so scripts are called by name:

script:
def args   = task.ext.args   ?: ''
def prefix = task.ext.prefix ?: "${meta.id}"
"""
my_script.py \\
    --input  ${input_file} \\
    --output ${prefix}.tsv \\
    ${args}
"""

Rules for bin/ scripts:

• Executable: chmod +x bin/my_script.py
• Shebang on line 1: #!/usr/bin/env python3 / #!/usr/bin/env Rscript / #!/usr/bin/env bash
• Dependencies declared in environment.yml and the container.
• Name after what it does, not after the module — one script may serve multiple modules.
• Keep command-line arguments explicit (--input, --output, thresholds, modes) so the module interface remains clear and easy to test.

---

Container selection

For Python-based local scripts, prefer Seqera Wave containers:

• Docker: community.wave.seqera.io/library/<pkg1>_<pkg2>:<hash>
• Singularity: https://community-cr-prod.seqera.io/docker/registry/v2/blobs/sha256/<hash>/data

Look up the Wave container for your dependency combination at https://seqera.io/wave/ or check existing local modules for a matching image. For non-Python tools, use the standard quay.io/biocontainers/ or depot.galaxyproject.org/singularity/ images.

---

meta.yml

Topic-channel modules use versions_<tool>: output sections and a topics: block instead of the old versions.yml output entry.

---
# yaml-language-server: $schema=https://raw.githubusercontent.com/nf-core/modules/master/modules/meta-schema.json
name: "module_name"
description: One sentence describing what the module does.
keywords:
  - keyword1
  - keyword2
tools:
  - "module_name":
      description: "Same as the top-level description."
      homepage: "<pipeline_repo_url>/tree/dev/modules/local/module_name/"
      documentation: "<pipeline_repo_url>/tree/dev/modules/local/module_name/"
      tool_dev_url: "<pipeline_repo_url>/tree/dev/modules/local/module_name/"
      doi: "<pipeline_doi_if_applicable>"
      licence: ["MIT"]
      identifier: ""
input:
  - - meta:
        type: map
        description: |
          Groovy Map containing sample information
          e.g. `[ id:'sample1' ]`
    - input_file:
        type: file
        description: Description of the input file.
        pattern: "*.{ext1,ext2}"
  - - threshold:
        type: float
        description: Description of the threshold parameter.
output:
  result:
    - - meta:
          type: map
          description: |
            Groovy Map containing sample information
            e.g. `[ id:'sample1' ]`
      - "*.ext":
          type: file
          description: Description of the output file.
          pattern: "*.ext"
  versions_tool_name:
    - - ${task.process}:
          type: string
          description: The process the versions were collected from
      - tool_name:
          type: string
          description: The tool name
      - tool --version 2>&1 | sed 's/tool //':
          type: eval
          description: The expression to obtain the version of the tool
topics:
  versions:
    # Repeat the corresponding versions_<tool> entries here.
authors:
  - "@vagkaratzas"
maintainers:
  - "@vagkaratzas"

Notes:

• Each input/output corresponds to one list item (the - - double-dash).
• Output channel names match the emit: names from main.nf.
• For multiple versioned tools, add one versions_<tool>: block per tool in output:, and one entry per tool in topics: versions:.

---

environment.yml

---
# yaml-language-server: $schema=https://raw.githubusercontent.com/nf-core/modules/master/modules/environment-schema.json
channels:
  - conda-forge
  - bioconda
dependencies:
  - python=3.13.1
  - conda-forge::biopython=1.85

Pin exact versions. Use the channel prefix (conda-forge::) only when the package doesn't come from the default channel order.

---

tests/main.nf.test

Always write at least two tests for each local module: one real run and one stub run. Treat tests and fixtures as required deliverables for complete local-module work, not a follow-up, unless the user explicitly says to skip tests.

nextflow_process {

    name "Test Process MODULE_NAME"
    script "../main.nf"
    process "MODULE_NAME"

    test("<dataset> - <input_type> - <output_type>") {

        when {
            process {
                """
                input[0] = channel.of([
                    [ id: 'test' ],
                    file("${moduleDir}/tests/fixtures/input.ext", checkIfExists: true)
                ])
                input[1] = 0.9
                """
            }
        }

        then {
            assertAll(
                { assert process.success },
                { assert snapshot(process.out).match() }
            )
        }
    }

    test("<dataset> - <input_type> - <output_type> - stub") {

        options "-stub"

        // identical when{} block as the real test above

        then {
            assertAll(
                { assert process.success },
                { assert snapshot(process.out).match() }
            )
        }
    }
}

Test naming

• Follow the nf-core pattern: "<dataset> - <input_type(s)> - <output_type>". Examples: "sarscov2 - fasta - alignment", "homo_sapiens - bam bai - vcf".
• Stub tests append " - stub" to the same name.
• Use a descriptive dataset name (e.g., sarscov2, homo_sapiens, or a short content descriptor like small_fasta).

Fixtures path

${moduleDir} in nf-test process tests resolves to the directory containing main.nf (i.e. modules/local/<name>/). Use it to reference local fixtures.

Assertion priority

Every snapshot must be inside assertAll() as snapshot(...).match(). Choose the strategy based on output stability — try option 1 first and only downgrade when it proves flaky:

1. Full snapshot (default — always try first):

{ assert snapshot(process.out).match() }

2. Per-channel + line count — when content varies but file length is stable:

{ assert snapshot(
    process.out.stable_channel,
    path(process.out.unstable_channel[0][1]).readLines().size(),
    process.out.findAll { key, val -> key.startsWith("versions") }
).match() }

3. Per-channel + line content match — when length also varies but a specific line is guaranteed:

{ assert snapshot(
    process.out.stable_channel,
    path(process.out.unstable_channel[0][1]).readLines().contains("<expected line>"),
    process.out.findAll { key, val -> key.startsWith("versions") }
).match() }

The .contains(...) returns a boolean that snapshots cleanly. Only use lines guaranteed by the tool's output spec, never incidental content.

4. Existence only (last resort):

{ assert snapshot(
    process.out.stable_channel,
    path(process.out.unstable_channel[0][1]).exists(),
    process.out.findAll { key, val -> key.startsWith("versions") }
).match() }

Rules:

• Stub tests always use option 1 (process.out) regardless of real test strategy.
• When using options 2–4, assert versions with process.out.findAll { key, val -> key.startsWith("versions") } — never path(process.out.versions[0]).yaml or any other form.
• Never snapshot empty-file md5sums (d41d8cd98f00b204e9800998ecf8427e). If a channel captures an empty file, the tool produced no real output — fix the test data or exclude that channel.

Environment setup

Ask the user which profile they use, then apply:

Profile	Requirement
singularity	Ask for NXF_SINGULARITY_CACHEDIR — do not assume a default; export NXF_SINGULARITY_CACHEDIR="<user path>"
docker	Docker daemon running — no extra env var
conda	conda on PATH — no extra env var

Test commands

CRITICAL — always prefix the profile with + (e.g. +singularity, +docker, +conda). The + appends the container profile on top of the base test profile. Omitting it replaces the base profile entirely, breaking nf-core test infrastructure.

# Run all tests
nf-test test /path/to/main.nf.test --profile +singularity --verbose

# Run a single test by name (faster iteration)
nf-test test /path/to/main.nf.test --profile +singularity --verbose --tag "<test_name>"

# Generate or update snapshot
nf-test test /path/to/main.nf.test --profile +singularity --verbose --update-snapshot

Replace +singularity with +docker or +conda as appropriate.

---

tests/fixtures/

Create the smallest real data file that exercises the module logic:

• FASTA: 3–5 short sequences (< 100 bp/aa each)
• TSV/CSV: 5–10 rows covering the range of values the module processes
• Compressed files: gzip the minimal file above

Place files directly in tests/fixtures/ and commit them alongside the module.

If a module has no natural file input, create the smallest fixture needed by downstream assertions or use inline scalar inputs in main.nf.test; still create a real and stub nf-test for the process.

---

Checklist before committing

• Process name is UPPER_SNAKE_CASE and matches directory name
• Every input tuple starts with val(meta)
• Version outputs use topic: versions — no versions.yml heredoc in script or stub
• meta.yml has versions_<tool>: output sections and a topics: versions: block (no versions.yml entry)
• Stub block only touches/creates output files — no script logic
• Simple one-line Bash/tool commands may stay inline; substantial custom Bash/Python/R logic lives in top-level bin/, unless the user explicitly requested inline module script bodies
• environment.yml has pinned versions
• If the module calls a bin/ script: file exists, has a shebang, and is executable (chmod +x)
• tests/fixtures/ contains minimal real data
• Every local module has tests/main.nf.test with one real run and one stub run, unless the user explicitly scoped tests out
• Snapshot strategy chosen per assertion priority (option 1 tried first); stubs always use process.out
• No empty-file md5sums (d41d8cd98f00b204e9800998ecf8427e) in snapshots
• Snapshot generated with --profile +<profile> --update-snapshot
• nextflow lint passes (not nf-core lint)