Structure reproducible bioinformatics projects with canonical layouts, restartable drivers, pinned environments, provenance, and lab notebooks. Use when starting or reorganizing a genomics project or making a sequencing analysis rerunnable.
التثبيت
التثبيت باستخدام Codex أو Claude انسخ هذا Prompt والصقه في Codex أو Claude أو مساعد آخر ليراجع صفحة Skill ويثبّتها لك.
Structure reproducible bioinformatics projects with canonical layouts, restartable drivers, pinned environments, provenance, and lab notebooks. Use when starting or reorganizing a genomics project or making a sequencing analysis rerunnable.
metadata
{"short-description":"Organize and reproducibly structure bioinformatics projects"}
Bioinformatics Project Organization
Set up a computational-biology project so a stranger can understand what was done and why, and so every analysis can be rerun end-to-end from a script. Apply this when creating, restructuring, or reproducibility-hardening any genomics, metagenomics, sequencing, or computational-biology project — and when running any single analysis command that should be reproducible.
Two assumptions drive every rule below (Noble 2009): a stranger must be able to reconstruct the analysis from the files alone, and everything will be rerun — with new data, new parameters, or a fixed bug — so build each experiment to re-execute from a script, never from memory.
Instructions
New project startup workflow
Before creating directories or running the first command, turn the project into
a small written contract:
Name the project with a durable, specific directory name.
Write the objective, analysis tracks, expected deliverables, input data
types, compute environment, and known reference databases in README.md.
Create tasks/METHODS.md immediately and record every setup command, software
version, database path, option, seed, and SLURM job ID as work proceeds.
Create SUMMARY.md for current status and high-level counts.
Create tasks/todo.md for active work and tasks/lessons.md for corrections
and prevention rules. When memd is available, initialize its project scope and
let memd memory-md maintain the root memory.md; do not hand-edit that file.
Initialize dependency management before analysis: use pixi.toml for
bioinformatics stacks, uv only for Python-only projects, and never rely on
system Python or ad hoc conda environments.
Check existing shared database locations before downloading references; if a
new reference is needed, put it in a named database directory and document
source URL, version, date, checksum, and command.
Put raw inputs and metadata in read-only data directories; never write
cleaned, mapped, filtered, or derived outputs back into raw data.
Before exploratory analysis, create a hypothesis register with at least five
biological, technical, null, sampling, and database explanations. Keep ruled-out
hypotheses visible with the evidence that changed their status.
For each analysis stage, create a restartable driver script plus a README
before launching compute, then update methods and summaries after the run.
Keep root-level directories intuitive from day one; do not let raw data,
generated batch outputs, publication figures, logs, and scratch files become
peers at the project root.
Canonical project layout
Logical at the root, chronological at the experiment level, logical again inside each experiment (Noble 2009). Cleaned and generated files go in results/, never back into data/ (Wilson 2017). For a simple single-track project, start here:
project_name/ # one self-contained, meaningfully named dir
├── data/ # fixed inputs + metadata; treat as read-only
│ ├── raw/ # immutable original data (chmod 444); never edit in place
│ └── README.md # provenance: source URL, version, download date, md5
├── results/ # all computed experiments (regenerable)
│ ├── lab_notebook.md # dated, verbose decision log at the root of results/
│ └── 2024-03-12_binning/ # one dated experiment dir (YYYY-MM-DD[_topic])
│ ├── runall # the driver script: reproduces this experiment alone
│ └── README.md # what/why for this experiment
├── src/ # your own analysis code (importable, tested)
├── bin/ # third-party / compiled tools, project wrappers
├── doc/ # one subdir per manuscript + notes + changelog
├── tasks/ # todo, methods, lessons, and hypothesis register
├── pixi.toml / pixi.lock # pinned bioinformatics tool stack (committed)
├── README.md LICENSE CITATION.cff
└── .gitignore # excludes data blobs and all results/ outputs
Date every experiment dir YYYY-MM-DD (optionally _topic) so they sort in time order — never final/ or binning_v2_really_final/. Full annotated tree, script categories, and the lab-notebook practice: references/project-layout.md (read when laying out or auditing a project). Concrete worked example: examples/project-tree.txt.
Numbered layout for multi-track projects
When one repository contains shared data plus multiple analyses, manuscripts,
or reusable preprocessing stages, prefer a numbered root layout. This keeps
shared inputs above project-specific tracks and prevents generated outputs from
accumulating at the root:
Use numbered directories for ordering, not as a substitute for names. Each
numbered directory still needs a short descriptive suffix. Put cross-track
inputs, sample manifests, read mappings, shared QC, and shared DuckDB databases
under 01_shared_preprocessing/. Put project-specific batch outputs and final
tables under the appropriate 02_analyses/<track>/ stage. Put manuscript
drafts, final figures, reports, and review packets under
03_publication_outputs/. Put SLURM stdout/stderr and temporary failed-tool
work directories under 90_logs/ or 99_scratch/, not beside final evidence.
For already-messy projects, do not one-shot move large output directories until
you have a path map. First inventory hard-coded paths, create the target
scaffold and README files, move data with temporary compatibility symlinks,
update scripts, then validate DuckDB builders, notebooks, and driver scripts
before removing old paths.
Reproducibility rules (non-negotiable)
Harden every analysis against Sandve 2013's ten rules: record provenance, script all steps (no manual edits), pin and archive exact tool versions, version-control all custom code, persist standard-format intermediates per stage, fix and record random seeds, store the raw data behind every plot, emit hierarchical drill-down output, tie every claim to its result in a literate document, and provide public access (deposit reads, assemblies, derived tables; push the repo with its lockfile). Each rule has a bioinformatics how-to with concrete commands in references/reproducibility-checklist.md — read it when hardening an analysis for reproducibility.
Driver-script discipline
Capture each experiment as one executable driver script, conventionally runall, so the whole analysis reproduces with one command (Noble 2009). Follow all six rules of thumb:
Record every operation — even gunzip and a one-off seqkit stats go in the script.
Comment generously — a reader understands the experiment from the comments alone.
Never hand-edit intermediates — transform with sed/awk/grep/cut so edits are recorded and repeatable.
Store all file/dir names as variables at the top — swapping a reference DB is a one-line change.
Use relative paths (../../data/...) so the project runs after checkout elsewhere.
Make it restartable — guard each step with "skip if output exists"; write outputs to a temp name then mv to the final name so a partial result is never mistaken for a complete one.
Pair runall with a summarize script (its final step) that produces a plot/table/HTML and can interpret a partially completed experiment. Abort on error (set -euo pipefail, check return codes, message to stderr, non-zero exit) and give every script a usage statement (Noble 2009). Runnable, idempotent template with version capture, fixed seed, and temp-then-rename: examples/runall.sh (copy and adapt when writing a pipeline).
Environment pinning
House rule for this repo: bioinformatics tool stacks use pixi (conda-forge + bioconda); Python-only deps use uv; never use system Python or conda directly.
Pin the stack in pixi.toml and commit pixi.lock; capture a container (Docker / Apptainer) for long-lived or shared analyses (Sandve 2013).
Record exact versions into each experiment's provenance file at run time (samtools --version, bwa 2>&1 | head, metabat2 2>&1 | head) and into tasks/METHODS.md.
Make dependencies machine-readable and explicit (Wilson 2017) — never rely on "whatever is on PATH".
Manifest example with lock-capture comment: examples/environment.pixi.toml. Full guidance on pixi, uv, conda export, and containers: references/environments.md (read when setting up or capturing an environment).
Version control
Put the project under Git for backup, history, and collaboration (Noble 2009); commit at least daily, keep changes small and focused (Wilson 2017).
Track only hand-edited files — code, configs, runall, notebook, sample sheets. Never commit generated outputs or binaries (*.bam, *.bai, *.fastq.gz, results/** tables); regenerate them via runall.
Use a .gitignore that ignores everything under results/ (including provenance.txt, which runall regenerates) and allow-lists the hand-edited files that live there — the driver, its README, the lab notebook. Tested pattern: examples/gitignore.example.
Branch for experimental work (git checkout -b try-metabat2); merge to main only when it works.
Tag the commit behind each published figure or release; archive the release for a DOI (Zenodo) (Sandve 2013, Wilson 2017).
Sharing & collaboration
Every project carries a README.md (study description, setup, how to reproduce), an explicit LICENSE, and a CITATION.cff; use tidy data (one variable per column, one observation per row) keyed by a stable unique ID (Wilson 2017). Detail: references/good-enough-practices.md (read for data management, software, collaboration, and manuscript practices).
Quick Reference
Need
Action
Start a simple project
Run scripts/scaffold_project.py with the default canonical layout, resolve pixi.lock, then replace the hypothesis placeholders.
Start a multi-track project
Add --layout numbered; use numbered, named roots for shared preprocessing, analyses, publication outputs, code, tests, logs, and scratch.
Add the first experiment
Add --first-experiment YYYY-MM-DD_topic to create a README and refusing runall template without overwriting project-owned files.
Add publication metadata
Opt in with --license MIT --author "Name or organization" --copyright-year YYYY; all three explicit values are required.
Repair a messy project
Inventory paths first, create the target scaffold, migrate with temporary compatibility links, and verify consumers before removing old paths.
Run an experiment
Copy examples/runall.sh, pin inputs and parameters, write atomically, and record versions and seeds.
Add structured metadata
Use bio-foundation-housekeeping as a separate follow-up only when schemas or a catalog are requested.
Prepare a release or deposition
Verify README, license, citation metadata, lockfiles, provenance, checksums, and regenerability.
Input Requirements
A project objective and expected deliverables.
Input data types, source locations, and available sample metadata.
Target compute environment, including scheduler constraints when applicable.
Known reference databases and their versions, or a plan to select and record them.
Existing path consumers when reorganizing a project.
Output
A documented project layout with immutable raw inputs and regenerable outputs.
tasks/todo.md, tasks/METHODS.md, tasks/lessons.md, and an exploratory hypothesis register.
A pinned pixi.toml and pixi.lock, or a uv lock for a Python-only project.
One restartable driver and README per analysis stage, with provenance, logs, and QC checks.
README, license, citation metadata, and deposition-ready provenance where sharing is in scope.
Quality Gates
Every generated result maps to a version-controlled driver and recorded command.
Raw inputs remain unchanged; derived data are written outside raw-data directories.
Tool and database versions, parameters, checksums, seeds, and scheduler job IDs are recorded.
Drivers fail on errors, write final outputs atomically, and resume without masking corrupt outputs.
Exploratory work starts with at least five hypotheses and records a reflection after each major QC gate.
A fresh checkout plus documented external inputs can reproduce the analysis.
No secrets, credentials, large generated outputs, or private data are staged for Git.
scripts/scaffold_project.py <project> --objective "..." --check passes, and a second identical scaffold run reports no created files.
When --first-experiment is used, its ISO date is valid and the generated runall exits non-zero until adapted.
Scaffolding never guesses a license or author; LICENSE/CITATION generation occurs only when license, author, and year are supplied together.
Examples
Use the bundled templates instead of recreating them:
examples/project-tree.txt for a concrete project tree.
examples/runall.sh for a restartable driver.
examples/environment.pixi.toml for the software environment.
examples/lab-notebook-entry.md for decision and result logging.
examples/gitignore.example for separating hand-edited files from generated outputs.
examples/samples.tsv for stable sample identifiers.
Troubleshooting
Existing paths are embedded in scripts or notebooks: Inventory every consumer before moving data. Add temporary compatibility symlinks, update consumers, and remove the links only after end-to-end validation.
A resume guard skips a corrupt output: Validate more than existence. Check nonzero size, parseability, expected record counts, checksums, or tool-specific completion markers; rerun corrupt stages with an explicit force option.
The environment cannot resolve: Confirm channel order and platform support in pixi.toml. Record the failed solve, then constrain the conflicting package or use a pinned container for that stage.
The repository needs schemas and a queryable catalog: Use bio-foundation-housekeeping after this skill establishes the project structure.
A scaffold file already contains different content: The command exits before writing anything and lists the conflicting paths. Reconcile those files manually; the scaffold does not overwrite project-owned content.
A first-experiment value is rejected: Use a real ISO calendar date plus a lowercase topic, for example 2026-07-11_read-qc. Slashes, .., uppercase topics, and impossible dates are rejected before the project directory is created.
Reference files
Load progressively as the task narrows:
references/project-layout.md — full annotated directory tree, dated results/ dirs, lab-notebook practice, script categories (Noble 2009). Read when laying out or auditing structure.
references/reproducibility-checklist.md — Sandve 2013's ten rules operationalized, each with a bioinformatics how-to. Read when making an analysis reproducible.
references/good-enough-practices.md — Wilson 2017 data/software/collaboration/org/tracking/manuscript practices. Read for sharing, licensing, tidy data, deposition.
references/environments.md — pinning and capturing environments with pixi, uv, conda, containers. Read when setting up or recording the software stack.
Example artifacts
scripts/scaffold_project.py — create or verify the minimal project scaffold without overwriting changed files.
examples/project-tree.txt — concrete annotated tree for an arctic metagenome / MAG-recovery study.
