Stale Script Remediation

Stale Script Remediation

Experiment Overview

Context

kintsugi workflow config only copies workflow/scripts/*.py to projects if they don't already exist. This means updates to scripts in the repo (e.g., adding signal isolation support to qc_report.py, updating log_utils.py with SI directory logging) never propagate to existing projects. This caused qc_signal_isolation failures in production (1901CC2A) and would affect all future processing.

The Problem

Detection method: MD5 comparison loop between repo source and deployed copies:

for proj_scripts in ../KINTSUGI_Projects/*/workflow/scripts/; do
    [ -d "$proj_scripts" ] || continue
    proj=$(basename "$(dirname "$(dirname "$proj_scripts")")")
    for script in deconvolve.py edf.py log_utils.py qc_report.py registration.py \
                  signal_isolation.py spillover_correction.py stitch.py vessel3d.py; do
        repo_md5=$(md5sum workflow/scripts/"$script" | awk '{print $1}')
        dep_md5=$(md5sum "$proj_scripts/$script" 2>/dev/null | awk '{print $1}')
        [ "$repo_md5" != "$dep_md5" ] && echo "STALE: $proj/$script"
    done
done

Verified Workflow

Step 1: Bulk-copy ALL scripts

for proj_scripts in ../KINTSUGI_Projects/*/workflow/scripts/; do
    [ -d "$proj_scripts" ] || continue
    for script in deconvolve.py edf.py log_utils.py qc_report.py registration.py \
                  signal_isolation.py spillover_correction.py stitch.py vessel3d.py; do
        cp workflow/scripts/"$script" "$proj_scripts/$script"
    done
done

Step 2: Update stale Snakefiles (if applicable)

for proj in src_CX_21-012_LN_n3 src_CX_21-012_SP-CC3-A ...; do
    cp workflow/Snakefile ../KINTSUGI_Projects/$proj/workflow/Snakefile
done

Step 3: Clear stale Snakemake locks

# First verify no snakemake processes running
ps aux | grep snakemake
squeue -u $USER

# Then clear locks
for proj in project1 project2; do
    rm -rf ../KINTSUGI_Projects/$proj/workflow/.snakemake/locks
    rm -rf ../KINTSUGI_Projects/$proj/.snakemake/locks
done

# Verify
find ../KINTSUGI_Projects -path '*/.snakemake/locks/*' -type f

Step 4: Verify with MD5 loop

Re-run the detection loop — should report Total stale scripts: 0.

Step 5: Launch batch processing

kintsugi workflow batch ../KINTSUGI_Projects --dry-run  # Confirm eligibility
kintsugi workflow batch ../KINTSUGI_Projects --detach -p 2

Failed Attempts (Critical)

Final Parameters

Key Insights

• workflow config is conservative by design — it protects user customizations by not overwriting existing scripts
• This creates a maintenance burden: every script update requires a bulk-copy pass
• The MD5 detection loop should be run periodically (or before batch launches) to catch drift
• Lock files in TWO locations: workflow/.snakemake/locks/ AND project-root/.snakemake/locks/
• Stale log_utils.py was universal (35/35) because it was updated after all projects were configured
• Snakefile staleness was limited to 6 empty projects with a different workflow architecture

Results

References

• workflow/CLAUDE.md — "Stale script deployment hazard" section
• MEMORY.md — "Stale Script Deployment & Bulk Remediation" section
• Skill old-code-data-detection — related pattern for detecting stale processed data