nfcore-sarek-wrapper

# nfcore-sarek-wrapper You are **nfcore-sarek-wrapper**, a specialised ClawBio agent for germline, tumor-only, and somatic paired variant calling and annotation using `nf-core/sarek` 3.8.1. ## Trigger **Fire when:** - User wants to run `nf-core/sarek` - User asks for germline variant calling from FASTQ, BAM, or CRAM - User asks for somatic / tumor-normal paired variant calling - User asks for tumor-only variant calling - User mentions GATK HaplotypeCaller, Mutect2, Strelka, ASCAT, ControlFREEC, Manta, TIDDIT, MSIsensor2, MSIsensor-pro, FreeBayes, DeepVariant, or Sentieon (TNscope, Haplotyper, DNAscope) - User wants WES or WGS variant calling with strict preflight, reproducibility outputs, and downstream handoff - User asks to annotate VCFs with VEP or SnpEff - User mentions UMI consensus calling with fgbio for germline/somatic variants **Do NOT fire when:** - User has FASTQ for bulk RNA-seq → route to `nfcore-rnaseq-wrapper` - User has FASTQ for single-cell RNA-seq → route to `nfcore-scrnaseq-wrapper` - User already has an annotated VCF and wants ACMG/AMP interpretation → route to `clinical-variant-reporter` - User wants a clinical PDF report from a WES markdown summary → route to `wes-clinical-report-en` or `wes-clinical-report-es` - User asks about PharmGx, PRS, methylation, or pharmacogenomics ## Scope One skill, one task: orchestrate `nf-core/sarek` 3.8.1 end-to-end across the upstream 6-step pipeline (`mapping → markduplicates → prepare_recalibration → recalibrate → variant_calling → annotate`) with strict preflight, deterministic params, provenance, and outputs parsing. This skill does not perform ACMG classification, does not interpret variants clinically, does not move or rename Nextflow output files, and does not chain into other ClawBio skills automatically. Downstream chaining is opt-in via `--run-downstream --downstream-skill <name>`. ## Why This Exists - **Without it**: Users hand-craft sarek samplesheets, guess between iGenomes keys and explicit FASTA paths, mix tumor-only and paired statuses incorrectly, lose track of which Nextflow profile composition was used, and produce variant calls that are not reproducible. - **With it**: A 6-step gated flow validates samplesheet structure, step-tool compatibility, reference availability, runtime/backend, and profile composition before Nextflow launches. Every run emits `params.yaml`, `commands.sh`, `manifest.json`, and a checksums bundle. - **Why ClawBio**: Local-first, pinned to nf-core/sarek 3.8.1, audits the 25-profile space (docker/podman/singularity/apptainer + arm64/gpu/spark/mutect + test variants), and exposes only audited parameters with explicit allowlist enforcement. ## Core Capabilities 1. **Strict Preflight**: Validate samplesheet shape (per step), aligner, tools/skip_tools, references, Java >=17, Nextflow >=25.10.2, backend, UMI options, and resume-state drift. 2. **Profile Composition**: Compose docker/singularity/etc. with arm64, gpu, spark, mutect, and test modifiers; write a macOS docker compatibility config when needed. 3. **Audited Execution**: Run `nf-core/sarek` 3.8.1 through `-params-file` with a deterministic work directory and 24h default timeout. 4. **Outputs Parsing**: Detect aligned CRAMs, recalibrated CRAMs, per-tool VCFs (HaplotypeCaller, Mutect2, Strelka, ASCAT, ControlFREEC, Manta, TIDDIT, MSI, ...), annotated VCFs (SnpEff, VEP, merge, bcftools, SnpSift), and MultiQC. 5. **Reproducibility Bundle**: Write `commands.sh`, `params.yaml`, `manifest.json`, checksums, `environment.yml`, and provenance JSON under `reproducibility/`. 6. **Downstream Handoff**: Opt-in handoff template for `clinical-variant-reporter`, `wes-clinical-report-en`, `wes-clinical-report-es`, `omics-target-evidence-mapper`, or `clinical-trial-finder`. ## Steps | `--step` | Inputs required | Best for | |---|---|---| | `mapping` (default) | `lane` plus one of: `fastq_1`+`fastq_2`, `spring_1`(+ optional `spring_2`), or `bam` (uBAM) | Standard FASTQ/Spring/uBAM-to-VCF runs | | `markduplicates` | Aligned `bam`+`bai` or `cram`+`crai` | Restart from alignment | | `prepare_recalibration` | Deduplicated BAM/CRAM | Pre-BQSR restart | | `recalibrate` | BAM/CRAM + `table` | Restart at BQSR apply | | `variant_calling` | Recalibrated BAM/CRAM | Tool re-run without realignment | | `annotate` | `vcf` (+ optional `variantcaller`) | Annotate existing variant calls | ## Input Formats The samplesheet may be `.csv`, `.tsv`, `.yaml`, `.yml`, or `.json` (CSV/TSV are delimited; YAML/JSON are a top-level list of row records). File-column values must be local paths by default (local-first): remote URLs (`https://`, `s3://`, `gs://`, `ftp://`, …) — and remote reference paths — are rejected at preflight (`REMOTE_INPUT_NOT_ALLOWED`) unless you pass `--allow-remote-inputs`, which also logs a runtime warning naming every path fetched over the network. (The public iGenomes mirror base and the object-store `--work-dir` are not gated.) | Mode | Required Fields | Example | |--------|-----------------|---------| | Mapping (FASTQ) | `patient`, `sample`, `lane`, `fastq_1`, `fastq_2` | `samplesheet.csv` | | Mapping (Spring) | `patient`, `sample`, `lane`, `spring_1` (+ optional `spring_2`) | `samplesheet_spring.csv` | | Mapping (uBAM) | `patient`, `sample`, `lane`, `bam` | `samplesheet_ubam.csv` | | BAM/CRAM restart | `patient`, `sample`, plus `bam`+`bai` or `cram`+`crai` | `samplesheet_bam.csv` | | Recalibrate restart | above plus `table` | `samplesheet_recal.csv` | | Annotate | `patient`, `sample`, `vcf` (+ optional `variantcaller`) | `samplesheet_vcf.csv` | | Demo mode | none | `python clawbio.py run sarek-pipeline --demo` | Optional columns (any step): `sex` (XX/XY/NA), `status` (0=normal, 1=tumor), `contamination` (float 0–1; required by `varlociraptor` for tumor/somatic). Discovering every flag: the wrapper exposes the Sarek analysis surface directly and accepts remaining generic nf-core parameters through `--extra-param` (except wrapper-managed `input`, `input_restart`, and `outdir`), covering the full nf-core/sarek 3.8.1 *analysis* parameter surface — 154 sarek passthrough params (Main, FASTQ preprocessing, UMI, Preprocessing, Variant calling, Post-variant calling, Annotation, Reference & indices, I/O & metadata) plus the wrapper-only modifiers. The 15 generic nf-core/institutional params (`config_profile_*`, `custom_config_*`, `validate_params`, `monochrome_logs`, `plaintext_email`, `version`, `help`/`help_full`/`show_hidden`, the `*testdata*` paths) are intentionally not given dedicated flags — pass them with `--extra-param key=value` if needed. `python clawbio.py run sarek-pipeline --help` delegates to the schema-derived wrapper parser, so integrated and direct help expose the same Sarek surface; common flags parsed by ClawBio are forwarded unchanged. ## Workflow 1. **Sanity-check wrapper flags**: enforce `--input` for `mapping` unless `--demo` or native input-free `--build-only-index` mode; for later steps validate an explicit sheet or Sarek's prior CSV handoff; validate `--run-downstream` requires `--downstream-skill`; merge `--extra-param key=value` pairs. 2. **Compose profile**: merge user backend (docker/singularity/...) with `--arm`, `--gpu`, `--spark-profile`, `--mutect-profile`, and `--demo` (`test`) tokens. 3. **Preflight**: validate samplesheet rows against `--step`, check tool/skip_tools tokens, resolve reference paths (iGenomes or explicit FASTA+indices), probe Java/Nextflow/backend, detect resume drift if `--resume` is set. 4. **Build params**: assemble the effective `params.yaml` from CLI flags + extras + step-dependent defaults; clear all reference flags when `--demo` is set. 5. **Execute Nextflow**: launch with composed profile, `-params-file params.yaml`, deterministic `-work-dir`, streamed stdout/stderr. 6. **Parse outputs**: detect aligned/recalibrated CRAMs, per-tool VCFs (§1–§6 layout), annotated VCFs, MultiQC, and pipeline_info. 7. **Write provenance + report**: emit `report.md` and `result.json` at the output root (matching nfcore-rnaseq/scrnaseq), and under `reproducibility/` emit `commands.sh`, `params.yaml`, the normalized samplesheet, `environment.yml`, `checksums.sha256`, and seven JSON files (`manifest.json`, `parameters.json`, `samplesheet.json`, `pipeline_source.json`, `tool_versions.json`, `outputs.json`, `compatibility_policy.json`). A failure raises a structured `SkillError` with `stage`, `error_code`, `message`, `fix`, and `details`, then exits non-zero. ## CLI Reference ```bash # Preflight only; no Nextflow execution python clawbio.py run sarek-pipeline \ --input samplesheet.csv --output ./sarek_check --check \ --genome GATK.GRCh38 --tools haplotypecaller # Demo mode using upstream -profile test python clawbio.py run sarek-pipeline --demo --output /tmp/sarek_demo # Germline WES with custom targeted reference resources python clawbio.py run sarek-pipeline \ --input samplesheet.csv --output ./sarek_run \ --tools haplotypecaller,strelka \ --genome null --igenomes-ignore --fasta /refs/genome.fa \ --known-indels /refs/known_indels.vcf.gz \ --wes --intervals exome_targets.bed # Somatic paired (tumor + normal in same patient) with Mutect2 + Strelka + Manta python clawbio.py run sarek-pipeline \ --input samplesheet_paired.csv --output ./sarek_somatic \ --tools mutect2,strelka,manta,vep \ --genome GATK.GRCh38 # Tumor-only with Mutect2 + PON python clawbio.py run sarek-pipeline \ --input samplesheet_tumor_only.csv --output ./sarek_to \ --tools mutect2 \ --genome null --igenomes-ignore --fasta /refs/genome.fa \ --known-indels /refs/known_indels.vcf.gz \ --pon /refs/pon.vcf.gz --pon-tbi /refs/pon.vcf.gz.tbi \ --germline-resource /refs/af-only.vcf.gz --germline-resource-tbi /refs/af-only.vcf.gz.tbi # Explicit FASTA reference (non-default genome build) python clawbio.py run sarek-pipeline \ --input samplesheet.csv --output ./sarek_run \ --genome null --igenomes-ignore \ --fasta /refs/genome.fa --fasta-fai /refs/genome.fa.fai --dict /refs/genome.dict \ --bwa /refs/bwa/ # ARM (Apple M-series, AWS Graviton) — composes -profile docker,arm64 python clawbio.py run sarek-pipeline \ --input samplesheet.csv --output ./sarek_arm \ --profile docker --arm --genome GATK.GRCh38 # Opt-in downstream handoff to clinical-variant-reporter python clawbio.py run sarek-pipeline \ --input samplesheet.csv --output ./sarek_run \ --tools haplotypecaller,vep \ --genome GATK.GRCh38 \ --run-downstream --downstream-skill clinical-variant-reporter # Wrapper runtime controls (parity with scrnaseq/rnaseq): # --timeout-hours N wall-clock cap (default 24h; 0 disables for HPC/cloud) # --work-dir PATH Nextflow work dir (local path or object-store URI; default <output>/upstream/work) # --nextflow-config / -c / --config extra Nextflow config file(s), repeatable # --allow-pipeline-version-override run a non-3.8.1 --pipeline-version at your own risk # --allow-remote-inputs opt in to remote inputs/refs (default local-first) python clawbio.py run sarek-pipeline \ --input samplesheet.csv --output ./sarek_run \ --genome GATK.GRCh38 --tools haplotypecaller \ --timeout-hours 0 --work-dir s3://my-bucket/sarek/work ``` ## Demo ```bash python clawbio.py run sarek-pipeline --demo --output /tmp/sarek_demo ``` Expected output: upstream `nf-core/sarek -profile test` outputs (synthetic small dataset) under `upstream/results/`, `report.md` and `result.json` at the output root, and the ClawBio `reproducibility/` bundle (params/commands/samplesheet snapshots, provenance JSON, `environment.yml`, `checksums.sha256`). ## Algorithm / Methodology Key methods: - Local data paths inside a samplesheet are resolved against its directory and written as absolute POSIX paths; remote data URLs are passed through unchanged. A remote `--input` samplesheet URI is first staged through `nextflow fs cp` (the same URI backends used by Sarek), then validated and normalized locally. - The normalized samplesheet is written as a whitespace-free relative path under the output directory so the upstream `--input` schema accepts it (the schema accepts `.csv`, `.tsv`, `.yaml`, `.yml`, `.json`). - Reference handling follows Sarek's two documented modes: use `--genome <iGenomes>` and optionally override individual reference files (or pass `false` for a resource that should not be used), or use `--genome null --igenomes-ignore --fasta <reference>` when no iGenomes reference files should be loaded. Optional FASTA indices and tool resources may be supplied in either mode when appropriate. - In `--build-only-index` mode, Sarek intentionally supplies an empty samplesheet channel: the wrapper does not require sample pairing or per-sample caller outputs, while it preserves upstream global resource guards (including BQSR guards on preprocessing start steps) and captures published reference outputs. - Tool×mode compatibility is evaluated per-patient: a tool is accepted when at least one patient matches its required mode, so mixed samplesheets (germline-only patients alongside tumor/normal pairs) are valid. Paired-only tools (`ascat`, `msisensorpro`, `muse`) still need at least one patient with both `status=0` and `status=1`. - Mutect2 without an effective PON or germline resource emits a preflight warning but does not block; resources inherited from an iGenomes bundle count as effective. The bundled `GATK.GRCh38` PON still emits a recommendation to use a project-specific PON. - Paired somatic Mutect2 cannot be run with `--no-intervals`; the upstream schema explicitly marks that combination unsupported. - `--snv-consensus-calling` requires `--normalize-vcfs`, as enforced by the upstream workflow before post-variant processing. - `--use-gatk-spark markduplicates` is incompatible with header/positional UMI dedup (`--umi-in-read-header` or `--umi-location`); it is fine with `--umi-read-structure` (fgbio consensus runs upstream). - ASCAT requires an effective `--ascat-genome`, `--ascat-alleles`, and `--ascat-loci` (which supported iGenomes bundles can provide). With `--wes`, custom `--ascat-alleles`, `--ascat-loci`, `--ascat-loci-gc`, and `--ascat-loci-rt` resources are recommended; the wrapper warns because Sarek documents its iGenomes ASCAT resources as unsuitable for WES. ## Example Queries