creating-analysis-projects

name	creating-analysis-projects
description	Use when setting up a new R or bioinformatics analysis project, scaffolding directory structures, refactoring messy scripts into clean pipelines, or establishing project conventions. Use when the user says "init", "set up a project", "refactor these scripts", or "create a pipeline". Also use when reviewing whether an existing project follows these conventions.

Creating Analysis Projects

Overview

Scaffold R/bioinformatics analysis projects with a strict separation between code (version-controlled) and data/outputs (not tracked). Git lives at the project root with a whitelist .gitignore that tracks only scripts/. Every notebook is self-contained. Every convention is enforced by structure, not discipline.

REQUIRED SUB-SKILLS:

• Use writing-r-code for all R code generation
• Use writing-qmd-scientific for all QMD document structure
• Use superpowers:brainstorming when designing a new pipeline from scratch
• Use superpowers:writing-plans when the pipeline has 3+ notebooks to implement

Directory Structure

The project uses a read / write / checkpoints triad as its core data flow architecture. This is non-negotiable — every project follows this layout:

project_name/                      # ← Git repo lives HERE
├── .git/
├── .gitignore                     # Whitelist: tracks only scripts/
├── read/                          # Raw input data — IMMUTABLE, never modified by code
│   └── reference/                 # Reference files (marker lists, GTFs, gene sets, etc.)
├── scripts/                       # ← All tracked code lives here
│   ├── .gitignore                 # Script-specific ignores (rendered outputs, caches, etc.)
│   ├── README.md
│   ├── project-name.Rproj         # RStudio project file (sets working directory)
│   ├── utils.R                    # Shared helper functions
│   ├── 01-first-step.qmd
│   ├── 02-second-step.qmd
│   └── ...
├── checkpoints/                   # RDS intermediates — FLAT, shared across scripts
│   ├── 01-merged-harmony.rds
│   └── 02-annotated.rds
└── write/                         # All pipeline outputs — PER-SCRIPT SUBDIRECTORIES
    ├── figures/
    │   ├── 01-first-step/         #   Subdirectory named after the qmd basename
    │   │   ├── 01-qc-violin.pdf   #   Each file still carries the step prefix
    │   │   └── 01-elbow-plot.pdf
    │   └── 02-second-step/
    │       └── 02-umap-condition.pdf
    └── tables/
        ├── 01-first-step/
        │   └── 01-qc-summary.csv
        └── 02-second-step/
            └── 02-condition-markers.csv

The read / write / checkpoints triad

These three directories define how data flows through the project:

• read/ — raw input data and reference files. Code reads from here but never writes to it. This directory is the immutable ground truth. Subdirectories organise by sample or data source (e.g., read/bmls-4/filtered_feature_bc_matrix/). Reference files like marker databases, gene annotations, and GTFs go in read/reference/.
• checkpoints/ — serialised R objects (.rds) that pass state between pipeline steps. Each notebook reads its predecessor's checkpoint and writes its own. These are the pipeline's internal handoff mechanism. Flat layout with step-prefixed filenames: 01-merged-harmony.rds, 02-annotated.rds. Stay flat because every downstream notebook reads checkpoints — subdirectories would obscure the shared-state nature. Within a notebook, intermediate checkpoints enable self-contained chunks: 04-fibro-subset.rds → 04-fibro-sct.rds → 04-fibro-harmony.rds → 04-fibro-states.rds.
• write/ — all human-facing outputs. Split into write/figures/ (plots saved via safe_ggsave()) and write/tables/ (CSVs, Excel files saved via readr::write_csv(), openxlsx). Per-script subdirectories: every notebook writes into its own folder named after the qmd basename (write/figures/<NN-script-slug>/, write/tables/<NN-script-slug>/), and the files inside still carry the matching NN- step prefix on their face. Dual provenance (folder + filename prefix) so a folder listing tells you which script produced everything, and an individual file copied elsewhere still self-identifies.

Why git at root with a scripts/ whitelist

• Git at root means one repo per project — no nested .git/ confusion, clean git clone into the project root
• The whitelist .gitignore (/* then !/scripts) means only scripts/ is ever tracked — everything else is silently ignored by default
• Adding new data directories (read/sample-2/, outputs/) never requires updating .gitignore
• read/ is immutable raw data — large, binary, doesn't belong in git
• checkpoints/ are ephemeral intermediates that change every re-run
• write/ contains generated outputs that are reproducible from code

Naming Conventions

File naming: hyphens

All filenames use hyphens (-): 01-qc-integration.qmd, 02-umap-singler.pdf, 04-fibro-states.rds

Code naming: underscores

All R variable and function names use snake_case (_): seu_integrated, marker_results_df, fix_md_rownames()

Output organisation

Outputs use a dual provenance scheme: a per-script subdirectory groups everything one notebook produced, and each file inside still carries the script's NN- step prefix so it self-identifies if copied elsewhere.

Per-script subdirectories for write/figures/ and write/tables/:

• The subdirectory name matches the qmd basename (without .qmd)
• Concrete example for scripts/02-tables-visualisations.qmd:
  • ../write/figures/02-tables-visualisations/02-hypoxia-marker-violins.pdf
  • ../write/figures/02-tables-visualisations/02-condition-marker-heatmap.pdf
  • ../write/tables/02-tables-visualisations/02-condition-markers-significant.csv
• The chunk that writes them must dir.create("../write/figures/02-tables-visualisations", recursive = TRUE, showWarnings = FALSE) first (see writing-r-code for the chunk template)

Step-prefixed filenames inside the subdirectory:

• Figures: 01-elbow-pca.pdf, 02-umap-singler.pdf
• Tables: 01-qc-summary.csv, 04-fibro-markers-all.csv

Checkpoints stay flat in checkpoints/ — never per-script subdirectories:

• 01-merged-harmony.rds, 02-annotated.rds, 04-fibro-states.rds
• Reason: downstream notebooks all read upstream checkpoints; per-script subdirs would force every reader to know which notebook wrote which file

Why dual provenance:

• The folder grouping makes "find every output from script 02" a one-line ls
• The filename prefix means an individual 02-condition-marker-heatmap.pdf copied into a paper draft or shared with a collaborator still announces its origin

Variable descriptiveness

Variable names must make their type and purpose obvious at a glance:

Bad	Good	Why
obj, seu	seu_integrated, seu_fibro	What Seurat object is this?
objs	seu_list	It's a list of Seurat objects
md	metadata_df	It's a metadata data frame
mat	expr_matrix	It's an expression matrix
p, p1	plot_umap_clusters, plot_violin_state	What does the plot show?
res	go_enrichment, marker_results_df	Results of what?
sid, i, ct	sample_name, cluster_id, cell_type	Loop iterators describe what they iterate
df, tab	composition_df, regulon_diagnostic_df	What data does it hold?

Keep it practical — 2-3 words max. seu_fibro not seurat_object_containing_fibroblast_subset.

Path Conventions

• All paths are relative from scripts/: ../read/, ../checkpoints/, ../write/figures/
• Paths appear inline at point of use — never defined as upfront variables
• Exception: external dependencies the user must provide (e.g., a GTF file path) get a clearly commented variable at the top of the relevant chunk

# Correct — inline at point of use, figures go in the per-script subdir
seu_integrated <- readr::read_rds("../checkpoints/01-merged-harmony.rds")
safe_ggsave("../write/figures/02-second-step/02-umap-singler.pdf",
            plot_umap, width = 8, height = 6)

# Incorrect — upfront path variables
checkpoint_dir <- "../checkpoints/"
fig_dir <- "../write/figures/"

# Incorrect — flat write/figures/ (no per-script subdir)
safe_ggsave("../write/figures/02-umap-singler.pdf", plot_umap, width = 8, height = 6)

Pipeline Structure

Numbered QMD notebooks

Each analysis step is a separate QMD file. Notebooks are numbered sequentially and pass data via RDS checkpoints:

01-first-step.qmd  →  ../checkpoints/01-output.rds
02-second-step.qmd →  reads 01-output.rds, writes ../checkpoints/02-output.rds
03-third-step.qmd  →  reads 02-output.rds, writes ../checkpoints/03-output.rds

Shared utils.R

Helper functions used across multiple notebooks go in utils.R:

• Every QMD sources it: source("utils.R")
• Functions are grouped by category with # Category ---------- headers
• No rm(list = ls()) or side effects — pure utility functions only

Self-contained code chunks

Every code chunk must be independently runnable. Follow the pattern from writing-r-code. Processing chunks save intermediate checkpoints; visualisation chunks load the latest checkpoint, dir.create() the per-script output subdirectory, and save figures into it.

Processing chunk (writes to flat checkpoints/):

# Libraries ----------
library(Seurat)
library(readr)

source("utils.R")

# Inputs ----------
seu <- readr::read_rds("../checkpoints/01-merged-harmony.rds")

# Processing ----------
seu <- NormalizeData(seu)

# Outputs ----------
readr::write_rds(seu, "../checkpoints/01-normalized.rds")

Visualisation chunk (writes to per-script subdir; chunk lives in 02-second-step.qmd):

# Libraries ----------
library(Seurat)
library(BadranSeq)
library(ggplot2)
library(readr)

source("utils.R")

# Inputs ----------
seu <- readr::read_rds("../checkpoints/01-normalized.rds")

dir.create("../write/figures/02-second-step",
           recursive = TRUE, showWarnings = FALSE)

# Processing ----------
plot_umap <- do_UmapPlot(seu, group.by = "celltype")

# Outputs ----------
ggsave("../write/figures/02-second-step/02-umap-celltype.pdf",
       plot_umap, width = 8, height = 6, bg = "white")

Prose format

Section descriptions use bullet points with bold key terms, never paragraphs:

## Quality Control

- **QC metrics**: `nFeature_RNA`, `nCount_RNA`, `percent.mt`
  - Mitochondrial fraction flags dying cells leaking cytoplasmic mRNA
- **Thresholds**: 200 ≤ nFeature ≤ 6000, nCount ≥ 500, percent.mt ≤ 20
  - Conservative but standard for 10x Chromium data
- **Rationale**: fixed thresholds chosen over MAD-based for reproducibility across batches

Version Control Setup

Initialise git at project root

cd project_name/
git init
git branch -m main

Root .gitignore — whitelist approach

Ignores everything by default; only scripts/ is tracked:

# Ignore everything
/*
/.*

# Whitelist: track only scripts/
!/scripts

Tracking empty data directories with .gitkeep

Git does not track empty directories. To make read/, checkpoints/, and write/ appear in the repo (so collaborators see the expected structure without committing any data), place an empty .gitkeep placeholder file in each:

touch read/.gitkeep checkpoints/.gitkeep write/figures/.gitkeep write/tables/.gitkeep

Then un-ignore only those placeholders in the root .gitignore:

# Ignore everything
*
*/

# Whitelist: tracked code and docs
!scripts/
!scripts/**
!docs/
!docs/**

# Expose .gitkeep placeholders so directory structure is visible in the repo
# (directory contents remain gitignored — data never touches the repo)
!read/.gitkeep
!checkpoints/.gitkeep
!write/figures/.gitkeep
!write/tables/.gitkeep

# Belt-and-braces: explicitly ignore directory contents even if rules above change
read/*
!read/.gitkeep
checkpoints/*
!checkpoints/.gitkeep
write/figures/*
!write/figures/.gitkeep
write/tables/*
!write/tables/.gitkeep

Why this works: .gitkeep is just an empty file — git has no special treatment for it. The name is a community convention. Once real data files land in the directory, .gitkeep sits there harmlessly and can be deleted if desired.

Why !*/ is required: The * rule ignores everything, including directories. When git hits an ignored directory it stops traversing into it entirely — so !read/.gitkeep alone would never fire because git would never enter read/ to see it. The !*/ rule un-ignores all directories globally, allowing git to walk into all of them. Once inside, read/* + !read/.gitkeep controls what gets tracked.

Why tracked scripts/ needs both !scripts/ and !scripts/**:

!*/ already lets git traverse scripts/, but it doesn't un-ignore the files inside. You still need:

!scripts/    # un-ignore the directory itself (redundant with !*/ but explicit)
!scripts/**  # un-ignore everything found inside at any depth

!scripts/** alone is not enough without !*/ or !scripts/ first — git won't enter an ignored directory. And * (single wildcard) only matches one level deep — ** is needed for recursive un-ignoring.

scripts/.gitignore — script-specific ignores

Place a second .gitignore inside scripts/ to exclude rendered outputs and caches:

# Rendered outputs
*.html
*_files/

# Quarto cache and intermediates
.quarto/
_freeze/
_site/

# R intermediates
*.Rhistory
.Rproj.user/
.RData
.Rapp.history

# Knitr cache
*_cache/

# OS files
.DS_Store
Thumbs.db

.Rproj template

Version: 1.0

RestoreWorkspace: No
SaveWorkspace: No
AlwaysSaveHistory: No

EnableCodeIndexing: Yes
UseSpacesForTab: Yes
NumSpacesForTab: 2
Encoding: UTF-8

AutoAppendNewline: Yes
StripTrailingWhitespace: Yes

README structure

The README must clearly separate tracked vs untracked content:

1. Project overview — biological/scientific context
2. Pipeline overview — DAG or numbered list of analysis steps
3. Per-step descriptions — what each notebook does
4. What is tracked — tree of repo contents with one-line descriptions
5. What is NOT tracked — tree of parent directory with setup instructions
6. Replication steps — exact git clone, mkdir, and data placement commands
7. Dependencies — R packages with categories
8. Code style — brief summary of conventions

GitHub repo creation

cd project_name/
git add scripts/
git commit -m "Initial commit: <N>-step <analysis-type> pipeline with shared utils"
gh repo create <descriptive-project-name> --private --source=. --push \
  --description "<one-line description of the analysis>"

Repo name should describe the science, not the tool: capsular-contracture-scrna not seurat-analysis-pipeline.

Scaffolding Checklist

When setting up a new project:

1. Create parent directory structure (read/, read/reference/, scripts/, checkpoints/, write/figures/, write/tables/)
2. Move/organise raw data into read/
3. Move reference files into read/reference/
4. Archive any existing scripts to .original_student_code/ or similar hidden directory
5. Create .Rproj in scripts/
6. Create utils.R in scripts/ with shared helpers
7. Create numbered QMD notebooks in scripts/
8. Initialise git at project root; create root .gitignore (whitelist) and scripts/.gitignore (script-specific ignores)
9. Write README.md
10. Create GitHub repo and push

Common Mistakes

Mistake	Correct Approach
git init inside scripts/	git init at project root with whitelist .gitignore
Defining path variables upfront	Paths inline at every function call
Paragraph prose in QMD sections	Bullet points with bold key terms
Generic variable names (obj, df, p)	Descriptive names (seu_fibro, composition_df, plot_umap_clusters)
Chunks that depend on previous chunk state	Every chunk loads its own inputs from disk
Output files without step prefix	Always prefix: 01-, 02-, etc.
Saving figures / tables flat under write/figures/ or write/tables/	Use per-script subdirs: write/figures/<NN-script-slug>/<NN-name>.pdf and write/tables/<NN-script-slug>/<NN-name>.csv
Forgetting to dir.create() the per-script output subdir	Each visualisation chunk runs dir.create("../write/figures/<NN-script-slug>", recursive = TRUE, showWarnings = FALSE) right after the # Inputs ---------- block
Per-script subdirs under checkpoints/	Checkpoints stay flat because downstream notebooks all read them: checkpoints/02-annotated.rds, not checkpoints/02-second-step/02-annotated.rds
readRDS()/saveRDS()	readr::read_rds()/readr::write_rds()
%>% magrittr pipe	|> native pipe
here::here() for paths	Relative ../ from scripts/
Mixing hyphens and underscores in filenames	Hyphens for files, underscores for code