Cycle Dir Normalization

Cycle Directory Normalization

Experiment Overview

Context

CODEX raw data often lands in long-form folders:

data/raw/cyc004_reg001_211206_201615/
data/raw/cyc005_reg001_211206_212937/

Both notebooks assume short form everywhere:

• Notebook 1 cell 14: cycle_folder = project.paths.raw / "cyc001" (hard-coded literal!)
• Notebook 2 cell 13 (raw preview): Path(image_dir) / f"cyc{cycle:03d}"
• Notebook 2 cell 16 (QC): glob(..., f'cyc{str(cycle).zfill(3)}', ...)
• Notebook 2 cell 42 (reprocess): f'cyc{REPROCESS_CYCLE:03d}'

Without normalization, these paths silently find zero files and the notebooks run "successfully" with empty results. An early version of Notebook 2 had a rename cell, but it was dropped (git history shows no explicit rename operations in any committed notebook version — likely existed only in a staging script that was never checked in).

The SLURM/Snakemake path works differently: workflow/scripts/stitch.py:find_cycle_dir() and Snakefile:_resolve_raw_cycle_dir() use glob patterns (cyc{N:03d}_*) to resolve long-form names at query time. That is why the pipeline batches ran fine on long-form data while the notebooks broke.

Verified Workflow

1. Shared helper in notebooks/Kio.py

_CYCLE_DIR_LONG_RE = re.compile(
    r"^(?:cyc|cycle)[_\-]?(\d+)(?:_reg\d+)?(?:[_\-].*)?$",
    re.IGNORECASE,
)

def normalize_cycle_dirs(raw_dir, width=3, dry_run=False, verbose=True):
    """Rename cyc<N>_reg<M>_... -> cyc<N zero-padded to width> in place."""
    raw_dir = Path(raw_dir)
    if not raw_dir.exists():
        return {}

    renames, collisions = {}, []
    for item in sorted(raw_dir.iterdir()):
        if not item.is_dir():
            continue
        m = _CYCLE_DIR_LONG_RE.match(item.name)
        if not m:
            continue
        short = f"cyc{int(m.group(1)):0{width}d}"
        if item.name == short:
            continue
        target = raw_dir / short
        if target.exists() and target.resolve() != item.resolve():
            collisions.append((item.name, short))
            continue
        renames[item.name] = short

    if collisions:
        raise RuntimeError(...)  # Never overwrite
    for old, new in renames.items():
        if not dry_run:
            (raw_dir / old).rename(raw_dir / new)
    return renames

2. Call before find_raw_cycles in both notebooks

Notebook 1 cell 5:

from Kio import normalize_cycle_dirs
normalize_cycle_dirs(project.raw_dir)
cycles = find_raw_cycles(project.raw_dir)

Notebook 2 cell 6: same pattern right before cycles = find_raw_cycles(image_dir).

3. Leave Snakemake/SLURM alone

workflow/scripts/stitch.py:find_cycle_dir() and Snakefile:_resolve_raw_cycle_dir() already handle long-form at glob time. Don't touch them — they're the fallback when the user runs the pipeline without ever opening the notebooks.

Failed Attempts (Critical)

Final Parameters

Key Insights

• Hard-coded path literals in notebook cells are a time bomb: they work on the datasets that were in the author's data/raw/ at development time, then silently fail when someone stages data with different naming.
• The SLURM/Snakemake path solving this via glob-at-query-time is elegant but doesn't help the notebooks, because notebooks don't call find_cycle_dir() from Kio — they build paths directly with f-strings.
• Post-commit sync hook (scripts/sync_to_projects.py) propagates Kio.py changes automatically to all 33+ project folders, so the fix lands everywhere on commit.
• When a user says "a cell was there in an early version" and git history doesn't show it, it was probably in a project-specific working copy that never got committed. Rebuild from first principles instead of hunting through branches.

References

• notebooks/Kio.py:334-436 — normalize_cycle_dirs() implementation
• notebooks/1_Single_Channel_Eval.ipynb cell 5 — call site
• notebooks/2_Cycle_Processing.ipynb cell 6 — call site
• workflow/scripts/stitch.py:154-175 — find_cycle_dir() (SLURM-side, glob-based, unchanged)
• workflow/Snakefile:86-112 — _resolve_raw_cycle_dir() (Snakemake-side, unchanged)

Related Skills

• notebook-module-refactoring — Same pattern (extract shared helpers to Kio.py); this skill is the "what to write" when the refactoring skill covers "how to extract safely"
• repo-project-sync-workflow — Edit in KINTSUGI/notebooks/ only; post-commit hook propagates