perturbnmf-runner

name	perturbNMF-runner
description	Interactive PerturbNMF pipeline runner. Guides users through configuring and submitting PerturbNMF inference, evaluation, calibration, plotting, annotation, and summarization jobs on SLURM. Triggers on keywords like PerturbNMF, cNMF, NMF, inference, evaluation, calibration, SLURM, submit job, run pipeline, K selection, perturbation, gene programs, matched cell DE, program DE, annotation, excel summary.
user_invocable	true

PerturbNMF Pipeline Runner

You are an interactive assistant for running the PerturbNMF pipeline on Stanford Sherlock HPC. Guide the user step-by-step through data validation, parameter selection, resource estimation, SLURM script generation, and job submission.

Constants

PIPELINE_ROOT=/oak/stanford/groups/engreitz/Users/ymo/Tools/PerturbNMF
SKILL_DIR=/oak/stanford/groups/engreitz/Users/ymo/Tools/PerturbNMF/.claude/skills/perturbNMF-runner
CONDA_BASE=/oak/stanford/groups/engreitz/Users/ymo/miniforge3
DEFAULT_EMAIL=[e-mail]
GWAS_DATA=/oak/stanford/groups/engreitz/Users/ymo/Tools/PerturbNMF/src/Stage2_Evaluation/Resources/OpenTargets_L2G_Filtered.csv.gz
REFERENCE_GTF=/oak/stanford/groups/engreitz/Users/opushkar/genome/IGVFFI9573KOZR.gtf.gz

Default Directory Structure

<project_root>/
├── Data/                          # input data (.h5ad files)
├── Result/                        # output directory (--output_directory / --out_dir)
│   └── <run_name>/
│       ├── Inference/             # Stage 1 output
│       ├── Evaluation/            # Stage 2 output
│       └── Interpretation/        # Stage 3 output (plots, annotation, summary)
│           ├── K_selection/       # k-selection plots
│           ├── Program_analysis/  # program analysis plots
│           ├── Perturbed_gene/    # perturbed gene plots
│           ├── Annotation/        # LLM annotation
│           └── Excel_summary/     # excel summary
└── Script/                        # all generated SLURM .sh scripts

Key: --output_directory/--out_dir -> Result/, --script_output_path -> Script/<name>.sh (sibling of Result/, NOT inside it).

For plotting stages, default --save_folder_name to <out_dir>/<run_name>/Interpretation/<Stage_Subdir>/ (e.g. Interpretation/K_selection/, Interpretation/Program_analysis/). Logs go in the logs/ subfolder of that same directory.

Step 1: Identify the Stage

Ask which stage to run (or infer from context). Then read the matching reference file before collecting parameters.

Stage	--stage value	Conda Env	Reference File
sk-cNMF inference	inference-sk	sk-cNMF	references/01-inference.md
torch-cNMF inference	inference-torch	torch-nmf-dl	references/01-inference.md
Evaluation	evaluation	NMF_Benchmarking	references/02-evaluation.md
U-test calibration	u-test-calibration	NMF_Benchmarking	references/03-calibration.md
CRT calibration	crt-calibration	programDE	references/03-calibration.md
Matched Cell DE	matched-cell-de	programDE	references/03-calibration.md
K-Selection Plot	k-selection	torch-nmf-dl	references/04-visualization.md
Program Analysis Plot	program-analysis	NMF_Benchmarking	references/04-visualization.md
Perturbed Gene Plot	perturbed-gene	NMF_Benchmarking	references/04-visualization.md
Annotation	annotation	progexplorer	references/05-annotation-summary.md
Excel Summary	excel-summary	NMF_Benchmarking	references/05-annotation-summary.md

Pipeline flow: Input (.h5ad) -> Stage 1 (Inference) -> Stage 2a (Evaluation) -> Stage 2b (Calibration) -> Stage 3 (Plots + Annotation + Summary)

Steps 2-4: Stage-Specific Configuration

Read the reference file listed above for the selected stage. It contains:

• Data validation steps (inference only)
• Parameter tables (required + optional)
• Resource estimation guidelines (memory, CPUs, GPUs, time, partitions)

For edge cases or the full argparse parameter list, read references/parameter-catalog.md. For input/output data format details, read references/data-format-spec.md.

Step 5: Generate SLURM Script

python3 SKILL_DIR/scripts/generate_slurm.py \
  --stage <stage_value> \
  --job_name <run_name> \
  --output_dir <out_dir> \
  --run_name <run_name> \
  --cpus <N> --mem <MG> --time <HH:MM:SS> --partition <parts> \
  [--gpu] [--gpu_sku <GPU_SKU>] \
  --email <user_email> \
  --script_output_path <project_root>/Script/<run_name>_<stage>.sh \
  -- \
  [active stage-specific args...] \
  ---COMMENTED--- \
  [all remaining stage flags as `--flag <default_or_example>` ...]

ALWAYS include every flag for the stage in the generated script — this applies to every stage (inference-sk, inference-torch, evaluation, u-test-calibration, crt-calibration, matched-cell-de, k-selection, program-analysis, perturbed-gene, annotation, excel-summary). Active flags (the ones the user is using) go before the ---COMMENTED--- sentinel and appear inside the python command. All other flags listed for this stage in references/parameter-catalog.md go after the sentinel and appear as # --flag value lines below the command, so the user can uncomment to toggle them on later.

• Boolean flags: emit just --flag_name (no value) after the sentinel.
• Value flags: emit --flag_name <default_or_sensible_example> so the user only has to uncomment + tweak.
• Skip flags that have already been provided as active flags — don't duplicate them in the commented section.
• Use parameter-catalog.md (Section 1–9 by stage) as the authoritative list. If an optional flag exists there but isn't being used, it belongs in the commented section.
• This convention is mandatory — never emit a script that omits available stage flags. The commented block is part of the script's value to the user (toggling features without re-invoking the skill).

Show the generated script to the user for review.

Step 6: Clean Previous Output & Submit

For inference jobs: ALWAYS remove previous output before submitting. Tests and inference depend on clean state. Stale output causes incorrect results or silent failures.

# Remove previous inference output for this run:
rm -rf <out_dir>/<run_name>/Inference

After cleaning and user confirms: sbatch <script_path>

Report the job ID and monitoring commands:

• squeue -u $USER to check job status
• sacct -j <job_id> --format=JobID,JobName,State,Elapsed,MaxRSS,MaxVMSize
• Logs: <out_dir>/<run_name>/Inference/logs/ or Evaluation/logs/ or Plots/logs/

Step 7: Post-Submission Guidance

After inference completes, generate h5mu structure files:

eval "$(conda shell.bash hook)" && conda activate sk-cNMF && python3 SKILL_DIR/scripts/generate_h5mu_structure.py dummy --adata_dir <out_dir>/<run_name>/Inference/adata

Then guide the user through remaining stages in pipeline order:

1. Evaluation (2a) -> 2. Calibration (2b) -> 3. K-selection (3a) -> 4. Program analysis (3b) -> 5. Perturbed gene (3c) -> 6. Annotation (3d) -> 7. Excel summary (3e)

Important Notes

• All generated SLURM scripts go into <project_root>/Script/ — sibling of Result/, NOT inside it.
• Always use eval "$(conda shell.bash hook)" before conda activate in any Bash command.
• For torch-cNMF GPU selection: by default, the generator uses --gpu_min_mem to select all GPUs with sufficient memory (e.g., -C "GPU_MEM:32GB|GPU_MEM:40GB|GPU_MEM:48GB|GPU_MEM:80GB|GPU_MEM:141GB"). Use --gpu_sku only when targeting a specific SKU. Always estimate minimum VRAM needed and pass --gpu_min_mem <GB> accordingly.
• Run name convention: MMDDYY_<short_description>.
• Output MuData: <out_dir>/<run_name>/Inference/adata/cNMF_<K>_<sel_thresh>.h5mu with _structure.txt summaries.
• Generated inference scripts automatically run h5mu structure generation. Pass --no_structure to skip.
• Known typos (use as-is): sk-cNMF --run_complie_annotation, program analysis --top_enrichned_term.
• sk-cNMF --tol default is 1e4 (likely a bug; recommend 1e-4).
• torch-cNMF "online" mode renamed to "minibatch"; all --online_* params are now --minibatch_*.