Creating Analysis Projects

Creating Analysis Projects

Overview

Scaffold R/bioinformatics analysis projects with a strict separation between code (version-controlled) and data/outputs (not tracked). Git lives at the project root with a whitelist .gitignore that tracks only scripts/. Every notebook is self-contained. Every convention is enforced by structure, not discipline.

REQUIRED SUB-SKILLS:

• Use writing-r-code for all R code generation
• Use writing-qmd-scientific for all QMD document structure
• Use superpowers:brainstorming when designing a new pipeline from scratch
• Use superpowers:writing-plans when the pipeline has 3+ notebooks to implement

Directory Structure

The project uses a read / write / checkpoints triad as its core data flow architecture. This is non-negotiable — every project follows this layout:

project-name/                      # ← Git repo lives HERE
├── .git/
├── .gitignore                     # Whitelist: tracks only scripts/
├── read/                          # Raw input data — IMMUTABLE, never modified by code
│   └── reference/                 # Reference files (marker lists, GTFs, gene sets, etc.)
├── scripts/                       # ← All tracked code lives here
│   ├── .gitignore                 # Script-specific ignores (rendered outputs, caches, etc.)
│   ├── README.md
│   ├── project-name.Rproj         # RStudio project file (sets working directory)
│   ├── utils.R                    # Shared helper functions
│   ├── 01-first-step.qmd
│   ├── 02-second-step.qmd
│   └── ...
├── checkpoints/                   # RDS intermediates — FLAT, shared across scripts
│   ├── 01-merged-harmony.rds
│   └── 02-annotated.rds
└── write/                         # All pipeline outputs — PER-SCRIPT SUBDIRECTORIES
    ├── figures/
    │   ├── 01-first-step/         #   Subdirectory named after the qmd basename
    │   │   ├── 01-qc-violin.pdf   #   Each file still carries the step prefix
    │   │   └── 01-elbow-plot.pdf
    │   └── 02-second-step/
    │       └── 02-umap-condition.pdf
    └── tables/
        ├── 01-first-step/
        │   └── 01-qc-summary.csv
        └── 02-second-step/
            └── 02-condition-markers.csv

The read / write / checkpoints triad

These three directories define how data flows through the project:

• read/ — raw input data and reference files. Code reads from here but never writes to it. This directory is the immutable ground truth. Subdirectories organise by sample or data source (e.g., read/bmls-4/filtered_feature_bc_matrix/). Reference files like marker databases, gene annotations, and GTFs go in read/reference/.
• checkpoints/ — serialised R objects (.rds) that pass state between pipeline steps. Each notebook reads its predecessor's checkpoint and writes its own. These are the pipeline's internal handoff mechanism. Flat layout with step-prefixed filenames: 01-merged-harmony.rds, 02-annotated.rds. Stay flat because every downstream notebook reads checkpoints — subdirectories would obscure the shared-state nature. Within a notebook, intermediate checkpoints enable self-contained chunks: 04-fibro-subset.rds → 04-fibro-sct.rds → 04-fibro-harmony.rds → 04-fibro-states.rds.
• write/ — all human-facing outputs. Split into write/figures/ (plots saved via safe_ggsave()) and write/tables/ (CSVs, Excel files saved via readr::write_csv(), openxlsx). Per-script subdirectories: every notebook writes into its own folder named after the qmd basename (write/figures/<NN-script-slug>/, write/tables/<NN-script-slug>/), and the files inside still carry the matching NN- step prefix on their face. Dual provenance (folder + filename prefix) so a folder listing tells you which script produced everything, and an individual file copied elsewhere still self-identifies.

Why git at root with a scripts/ whitelist

• Git at root means one repo per project — no nested .git/ confusion, clean git clone into the project root
• The whitelist .gitignore (/* then !/scripts) means only scripts/ is ever tracked — everything else is silently ignored by default
• Adding new data directories (read/sample-2/, outputs/) never requires updating .gitignore
• read/ is immutable raw data — large, binary, doesn't belong in git
• checkpoints/ are ephemeral intermediates that change every re-run
• write/ contains generated outputs that are reproducible from code

Naming Conventions

File naming: hyphens

All filenames use hyphens (-): 01-qc-integration.qmd, 02-umap-singler.pdf, 04-fibro-states.rds

Code naming: underscores

All R variable and function names use snake_case (_): seu_integrated, marker_results_df, fix_md_rownames()

Underscores are for code identifiers only — never for filenames or directory names. naming-conventions holds the project-wide rule and the dated-asset pattern.

Output organisation

Outputs use a dual provenance scheme: a per-script subdirectory groups everything one notebook produced, and each file inside still carries the script's NN- step prefix so it self-identifies if copied elsewhere.

Per-script subdirectories for write/figures/ and write/tables/:

• The subdirectory name matches the qmd basename (without .qmd)
• Concrete example for scripts/02-tables-visualisations.qmd:
  • ../write/figures/02-tables-visualisations/02-hypoxia-marker-violins.pdf
  • ../write/figures/02-tables-visualisations/02-condition-marker-heatmap.pdf
  • ../write/tables/02-tables-visualisations/02-condition-markers-significant.csv
• The script's setup chunk dir.create()s these directories once, at the top of the document (see writing-r-code for the setup chunk template)

Step-prefixed filenames inside the subdirectory:

• Figures: 01-elbow-pca.pdf, 02-umap-singler.pdf
• Tables: 01-qc-summary.csv, 04-fibro-markers-all.csv

Checkpoints stay flat in checkpoints/ — never per-script subdirectories:

• 01-merged-harmony.rds, 02-annotated.rds, 04-fibro-states.rds
• Reason: downstream notebooks all read upstream checkpoints; per-script subdirs would force every reader to know which notebook wrote which file

Why dual provenance:

• The folder grouping makes "find every output from script 02" a one-line ls
• The filename prefix means an individual 02-condition-marker-heatmap.pdf copied into a paper draft or shared with a collaborator still announces its origin

Variable descriptiveness

Variable names must make their type and purpose obvious at a glance:

Keep it practical — 2-3 words max. seu_fibro not seurat_object_containing_fibroblast_subset.

Path Conventions

• All paths are relative from scripts/: ../read/, ../checkpoints/, ../write/figures/
• Paths appear inline at point of use — never defined as upfront variables
• Exception: external dependencies the user must provide (e.g., a GTF file path) get a clearly commented variable at the top of the relevant chunk

# Correct — inline at point of use, figures go in the per-script subdir
seu_integrated <- readr::read_rds("../checkpoints/01-merged-harmony.rds")
safe_ggsave("../write/figures/02-second-step/02-umap-singler.pdf",
            plot_umap, width = 8, height = 6)

# Incorrect — upfront path variables
checkpoint_dir <- "../checkpoints/"
fig_dir <- "../write/figures/"

# Incorrect — flat write/figures/ (no per-script subdir)
safe_ggsave("../write/figures/02-umap-singler.pdf", plot_umap, width = 8, height = 6)

Pipeline Structure

Numbered QMD notebooks

Each analysis step is a separate QMD file. Notebooks are numbered sequentially and pass data via RDS checkpoints:

01-first-step.qmd  →  ../checkpoints/01-output.rds
02-second-step.qmd →  reads 01-output.rds, writes ../checkpoints/02-output.rds
03-third-step.qmd  →  reads 02-output.rds, writes ../checkpoints/03-output.rds

Shared utils.R

Helper functions used across multiple notebooks go in utils.R:

• Every QMD sources it: source("utils.R")
• Functions are grouped by category with # Category ---------- headers
• No rm(list = ls()) or side effects — pure utility functions only

Setup chunk

Every notebook opens with one setup chunk, under its own ## Setup heading, holding the conflicted preferences and every output directory the script writes to. See writing-r-code for the full template.

# Libraries ----------
library(conflicted)

# Conflicts ----------
conflicts_prefer(
  dplyr::filter, dplyr::count, dplyr::rename, dplyr::slice,
  purrr::reduce,
  base::intersect, base::setdiff, base::union, base::setequal,
  .quiet = TRUE
)

# Output directories ----------
dir.create("../checkpoints", recursive = TRUE, showWarnings = FALSE)

dir.create("../write/figures/02-second-step",
           recursive = TRUE, showWarnings = FALSE)

dir.create("../write/tables/02-second-step",
           recursive = TRUE, showWarnings = FALSE)

Self-contained code chunks

Every code chunk must be independently runnable once the setup chunk has run. Follow the pattern from writing-r-code. Processing chunks save intermediate checkpoints; visualisation chunks load the latest checkpoint and save figures into the per-script output subdirectory that setup already created.

Processing chunk (writes to flat checkpoints/):

# Libraries ----------
library(Seurat)
library(readr)

source("utils.R")

# Inputs ----------
seu <- readr::read_rds("../checkpoints/01-merged-harmony.rds")

# Processing ----------
seu <- NormalizeData(seu)

# Outputs ----------
readr::write_rds(seu, "../checkpoints/01-normalized.rds")

Visualisation chunk (writes to per-script subdir; chunk lives in 02-second-step.qmd):

# Libraries ----------
library(Seurat)
library(BadranSeq)
library(ggplot2)
library(readr)

source("utils.R")

# Inputs ----------
seu <- readr::read_rds("../checkpoints/01-normalized.rds")

# Processing ----------
plot_umap <- do_UmapPlot(seu, group.by = "celltype")

# Outputs ----------
ggsave("../write/figures/02-second-step/02-umap-celltype.pdf",
       plot_umap, width = 8, height = 6, bg = "white")

Prose format

Section descriptions use bullet points with bold key terms, never paragraphs:

## Quality Control

- **QC metrics**: `nFeature_RNA`, `nCount_RNA`, `percent.mt`
  - Mitochondrial fraction flags dying cells leaking cytoplasmic mRNA
- **Thresholds**: 200 ≤ nFeature ≤ 6000, nCount ≥ 500, percent.mt ≤ 20
  - Conservative but standard for 10x Chromium data
- **Rationale**: fixed thresholds chosen over MAD-based for reproducibility across batches

Version Control Setup

Initialise git at project root

cd project-name/
git init
git branch -m main

Root .gitignore — whitelist approach

Ignores everything by default; only scripts/ is tracked:

# Ignore everything
/*
/.*

# Whitelist: track only scripts/
!/scripts

Tracking empty data directories with .gitkeep

Git does not track empty directories. To make read/, checkpoints/, and write/ appear in the repo (so collaborators see the expected structure without committing any data), place an empty .gitkeep placeholder file in each:

touch read/.gitkeep checkpoints/.gitkeep write/figures/.gitkeep write/tables/.gitkeep

Then un-ignore only those placeholders in the root .gitignore:

# Ignore everything
*
*/

# Whitelist: tracked code and docs
!scripts/
!scripts/**
!docs/
!docs/**

# Expose .gitkeep placeholders so directory structure is visible in the repo
# (directory contents remain gitignored — data never touches the repo)
!read/.gitkeep
!checkpoints/.gitkeep
!write/figures/.gitkeep
!write/tables/.gitkeep

# Belt-and-braces: explicitly ignore directory contents even if rules above change
read/*
!read/.gitkeep
checkpoints/*
!checkpoints/.gitkeep
write/figures/*
!write/figures/.gitkeep
write/tables/*
!write/tables/.gitkeep

Why this works: .gitkeep is just an empty file — git has no special treatment for it. The name is a community convention. Once real data files land in the directory, .gitkeep sits there harmlessly and can be deleted if desired.

Why !*/ is required: The * rule ignores everything, including directories. When git hits an ignored directory it stops traversing into it entirely — so !read/.gitkeep alone would never fire because git would never enter read/ to see it. The !*/ rule un-ignores all directories globally, allowing git to walk into all of them. Once inside, read/* + !read/.gitkeep controls what gets tracked.

Why tracked scripts/ needs both !scripts/ and !scripts/**:

!*/ already lets git traverse scripts/, but it doesn't un-ignore the files inside. You still need:

!scripts/    # un-ignore the directory itself (redundant with !*/ but explicit)
!scripts/**  # un-ignore everything found inside at any depth

!scripts/** alone is not enough without !*/ or !scripts/ first — git won't enter an ignored directory. And * (single wildcard) only matches one level deep — ** is needed for recursive un-ignoring.

scripts/.gitignore — script-specific ignores

Place a second .gitignore inside scripts/ to exclude rendered outputs and caches:

# Rendered outputs
*.html
*_files/

# Quarto cache and intermediates
.quarto/
_freeze/
_site/

# R intermediates
*.Rhistory
.Rproj.user/
.RData
.Rapp.history

# Knitr cache
*_cache/

# OS files
.DS_Store
Thumbs.db

.Rproj template

Version: 1.0

RestoreWorkspace: No
SaveWorkspace: No
AlwaysSaveHistory: No

EnableCodeIndexing: Yes
UseSpacesForTab: Yes
NumSpacesForTab: 2
Encoding: UTF-8

AutoAppendNewline: Yes
StripTrailingWhitespace: Yes

README structure

The README must clearly separate tracked vs untracked content:

1. Project overview — biological/scientific context
2. Pipeline overview — DAG or numbered list of analysis steps
3. Per-step descriptions — what each notebook does
4. What is tracked — tree of repo contents with one-line descriptions
5. What is NOT tracked — tree of parent directory with setup instructions
6. Replication steps — exact git clone, mkdir, and data placement commands
7. Dependencies — R packages with categories
8. Code style — brief summary of conventions

GitHub repo creation

cd project-name/
git add scripts/
git commit -m "Initial commit: <N>-step <analysis-type> pipeline with shared utils"
gh repo create <descriptive-project-name> --private --source=. --push \
  --description "<one-line description of the analysis>"

Repo name should describe the science, not the tool: capsular-contracture-scrna not seurat-analysis-pipeline.

Scaffolding Checklist

When setting up a new project:

1. Create parent directory structure (read/, read/reference/, scripts/, checkpoints/, write/figures/, write/tables/)
2. Move/organise raw data into read/
3. Move reference files into read/reference/
4. Archive any existing scripts to .original_student_code/ or similar hidden directory
5. Create .Rproj in scripts/
6. Create utils.R in scripts/ with shared helpers
7. Create numbered QMD notebooks in scripts/, each opening with a ## Setup section and its setup chunk
8. Initialise git at project root; create root .gitignore (whitelist) and scripts/.gitignore (script-specific ignores)
9. Write README.md
10. Create GitHub repo and push

Common Mistakes