Batch Isolation

Batch Signal Isolation with Recipe-Driven Processing

Goal

Replace naive batch signal isolation with a recipe-driven pipeline that matches the legacy Notebook 3 interactive workflow: multi-step subtraction, per-marker tuned parameters, background cleaning, and parameter learning for cross-dataset transfer.

Context

CX_19-001_SP_CC2-A28 (13 cycles, 28 signal markers, spleen) was first processed with auto-analyzed parameters at ~8 min/channel. The results were poor compared to the manually tuned Notebook 3 outputs — the batch code only did a single subtraction step with expensive blank smoothing (sigma=500), while the legacy pipeline uses primary + secondary subtraction + background cleaning with per-marker tuned parameters.

Root cause: Our batch code implemented a single subtraction with automated parameters. The legacy pipeline uses a multi-step process:

1. Primary subtraction with tuned clip/scale factors
2. Optional second subtraction with a different blank
3. Background cleaning (threshold + median smooth + small object removal)

What Worked

Recipe-Driven Pipeline (v2.0)

load_legacy_recipes(recipe_dir) parses *_param.txt files (key-value format from Notebook 3) into MarkerRecipe dataclasses:

• Uses ast.literal_eval() with fallback for unparseable values
• Skips dask.array<...> and datetime.datetime(...) artifact lines
• Handles missing fields with dataclass defaults

Three new dataclasses: SubtractionParams, CleanParams, MarkerRecipe

Blank name resolution chain (4 steps):

1. _normalize_blank_name(): Blank1b → Blank_1b, Blank13c → Blank_13c
2. Exact match in location map (all channel names → paths)
3. Fuzzy match: strip hyphens/underscores, case-insensitive (e.g., HLADR → HLA-DR)
4. Positional fallback: _extract_blank_position() gets suffix (a/b/c), tries Blank_1{suffix} — remaps higher-cycle blanks (e.g., Blank_13b → Blank_1b) when target project only has cycle 1 blanks

clean_background(image, params) — pure numpy reimplementation of Kutils.clean():

1. Zero pixels below background_threshold
2. Median filter in transition zone (between 0 and smooth_threshold)
3. remove_small_objects + morphological closing

Performance: sigma=0 + recipes = ~30 sec/channel

• Old: ~8 min/channel (sigma=500 blank smoothing dominated)
• New with recipes + sigma=0: ~30 sec/channel
• Full batch (28 markers): completed successfully, all 28 channels processed

Auto Method Selection (unchanged from v1)

• Rule 1: blank_p99 / signal_p99 > 1.2 → weighted
• Rule 2: af_contribution > 0.3 AND correlation > 0.4 → weighted
• Rule 3: correlation > 0.5 AND dynamic_range > 5000 → weighted
• Default: global

Parameter Learning DB Integration

• Records recipe outcomes to ParameterLearningEngine after successful processing
• Two operations per channel: blank_subtraction and clean_background
• algorithm_version="recipe_v1" distinguishes from auto-analyzed params
• --learn/--no-learn CLI flag controls recording (default: --learn)
• Enables cross-dataset transfer: manually tuned params from CX_19-001 spleen propagate to other spleen datasets

Self-Normalized QC (unchanged from v1)

• Before/After/Difference 3-column layout
• Self-normalized to own p1-p99 — dim signal visible instead of all-black

CLI Updates

• --recipe-dir option on plan and run subcommands
• --learn/--no-learn flag on run subcommand
• --tile-smooth-sigma default changed from 500 to 0
• Summary output includes recipe count

Failed Attempts

Key Files

Final Parameters

# Recipe-driven processing (default path with --recipe-dir)
# No blank smoothing — recipes provide exact parameters
tile_smooth_sigma = 0.0

# Quality gate (over-subtraction detection + auto fallback)
quality_gate = 0.6          # Minimum quality score; 0 to disable
clip_percentile = 99.5      # Outlier clipping; 0 to disable

# Method selection thresholds (for markers without recipes)
BLANK_DOMINANCE_RATIO = 1.2
AF_CONTRIBUTION_THRESHOLD = 0.3
CORRELATION_THRESHOLD_AF = 0.4
CORRELATION_THRESHOLD_RANGE = 0.5
DYNAMIC_RANGE_THRESHOLD = 5000

# QC visualization
page_size = 6
downsample = 16
dpi = 120
p_low, p_high = 1.0, 99.0

# CLI usage
# kintsugi workflow isolate run . --recipe-dir .../Processing_parameters --tissue-type spleen
# kintsugi workflow isolate run . --recipe-dir .../Processing_parameters --channels CD3e --force
# kintsugi workflow isolate qc .

Snakemake Pipeline Integration (Feb 2026)

Signal isolation is now Rule 5 in the Snakemake pipeline, running automatically after registration. The process_batch() function is called by workflow/scripts/signal_isolation.py as a Snakemake wrapper script. This eliminates manual CLI chaining — kintsugi workflow batch now runs the full pipeline through signal isolation. See snakemake-signal-isolation skill for implementation details.

Environment

• Python 3.11, NumPy <2.0, SciPy, scikit-image, tifffile, matplotlib, PyYAML
• HiPerGator SLURM cluster (login nodes, no GPU required for signal isolation)
• Images: ~189 MB per channel (7479x12662 px), uint16
• Processing with recipes: ~30 sec/channel (was ~8 min with sigma=500)
• Test suite: 94 tests, <7 seconds

Verified On

• CX_19-001_SP_CC2-A28: 13 cycles, 28 signal markers, spleen tissue
  • 24 recipe-driven + 4 auto-weighted (CD5, HLA-DR, PanCK, Podoplanin)
  • Mean quality score: 0.803, 0 errors
• Synthetic test data: 256x256 and 64x64 images with various channel configurations