Exp Lens Iterative Learning

Iterative Learning Experimental Design Lens

Philosophical Mode: Decision-Theoretic Primary Question: "How does this maximize learning per cost?" Focus: Factor Selection, Interaction Probing, Adaptive Allocation, Stopping Rules, Next-Experiment Planning

When to Use

• Planning a sequence of experiments
• Optimizing hyperparameter search
• Ablation study design
• User invokes /exp-lens-iterative-learning or /make-experiment-diag iterative

Critical Constraints

NEVER:

• Modify any source code files
• Do not litter the codebase with useless comments, TODO markers, or explanatory annotations — the skill output and diagram speak for themselves
• Recommend one-factor-at-a-time exploration when interactions are plausible

ALWAYS:

• Evaluate exploration efficiency against the key uncertainty being reduced
• Identify high-value unexplored regions of the factor space
• Assess whether the stopping rule is principled or arbitrary
• Surface interaction structure that one-factor-at-a-time designs would miss
• BEFORE creating any diagram, LOAD the /mermaid skill using the Skill tool - this is MANDATORY

Analysis Workflow

Step 1: Launch Parallel Exploration Subagents

Spawn Explore subagents to investigate:

Factor Space

• Find all experimental factors and their ranges
• Look for: hyperparameter, config, param, factor, level, range, grid, sweep

Interaction Structure

• Find evidence of factor interactions
• Look for: interaction, ablation, factorial, combination, cross, joint

Cost & Resource Model

• Find compute/time/cost constraints
• Look for: budget, cost, time, gpu, compute, resource, quota, limit

Sequential Decision Logic

• Find adaptive or sequential experiment logic
• Look for: bayesian_opt, optuna, hyperband, successive, early_stop, adaptive, bandit

Learning Objectives

• Find what uncertainty the experiments aim to reduce
• Look for: objective, goal, hypothesis, question, target, uncertainty, explore

Step 2: Map the Design Space

Map the design space (factors x levels), identify which regions have been explored, which interactions are probed, and what the next high-value experiment would be. Assess whether the experimental sequence is efficient (reducing key uncertainty) or wasteful (one-factor-at-a-time in interacting systems).

Step 3: CRITICAL — Analyze Learning Efficiency

For every factor and experiment round:

• Information gain: What uncertainty does this experiment reduce? What does it leave unresolved?
• Interaction risk: Are there factors that likely interact? Does the design probe those interactions?
• Cost-efficiency: What is the cost per unit of learning? Is there a cheaper design with equivalent learning value?
• Exploration-exploitation: Is the strategy exploring new regions or exploiting known good regions prematurely?
• Stopping rule: Is the stopping criterion based on a principled uncertainty threshold, or is it arbitrary (fixed budget, time limit)?

Distinguish clearly:

• Full factorial: Explores all combinations — exhaustive but expensive, catches all interactions
• Fractional factorial: Efficient but aliases some interaction effects
• One-factor-at-a-time: Simple but misses interactions — inefficient in correlated spaces
• Adaptive/Bayesian: High efficiency when the response surface is smooth — risks local optima

Step 4: Create the Diagram

Use flowchart with:

Direction: LR (learning flows from exploration to refinement)

Subgraphs:

• FACTOR SPACE
• EXPLORATION STRATEGY
• RESULTS SO FAR
• NEXT EXPERIMENTS
• STOPPING CRITERIA

Node Styling:

• stateNode class: factors/parameters
• handler class: exploration strategies
• phase class: experiment rounds
• output class: results/learnings
• cli class: objectives
• gap class: unexplored high-value regions
• detector class: stopping criteria

Step 5: Write Output

Write the diagram to: temp/exp-lens-iterative-learning/exp_diag_iterative_learning_{YYYY-MM-DD_HHMMSS}.md

Output Template

# Iterative Learning Diagram: {System Name}

**Lens:** Iterative Learning (Decision-Theoretic)
**Question:** How does this maximize learning per cost?
**Date:** {YYYY-MM-DD}
**Scope:** {What was analyzed}

## Factor Space

| Factor | Range | Levels | Explored? | Interaction Risk |
|--------|-------|--------|-----------|-----------------|
| {Factor 1} | {min}–{max} | {n} | ✓ Full | Low |
| {Factor 2} | {min}–{max} | {n} | ✓ Partial | High |
| {Factor 3} | {min}–{max} | {n} | ✗ None | Unknown |

## Iterative Learning Design-Space Diagram

```mermaid
%%{init: {'flowchart': {'nodeSpacing': 50, 'rankSpacing': 60, 'curve': 'basis'}}}%%
flowchart LR
    %% CLASS DEFINITIONS %%
    classDef cli fill:#1a237e,stroke:#7986cb,stroke-width:2px,color:#fff;
    classDef stateNode fill:#004d40,stroke:#4db6ac,stroke-width:2px,color:#fff;
    classDef handler fill:#e65100,stroke:#ffb74d,stroke-width:2px,color:#fff;
    classDef phase fill:#6a1b9a,stroke:#ba68c8,stroke-width:2px,color:#fff;
    classDef newComponent fill:#2e7d32,stroke:#81c784,stroke-width:2px,color:#fff;
    classDef output fill:#00695c,stroke:#4db6ac,stroke-width:2px,color:#fff;
    classDef detector fill:#b71c1c,stroke:#ef5350,stroke-width:2px,color:#fff;
    classDef gap fill:#ff6f00,stroke:#ffa726,stroke-width:2px,color:#000;
    classDef integration fill:#c62828,stroke:#ef9a9a,stroke-width:2px,color:#fff;

    subgraph Factors ["FACTOR SPACE"]
        direction TB
        F1["{Factor 1}<br/>━━━━━━━━━━<br/>{range}, {levels} levels"]
        F2["{Factor 2}<br/>━━━━━━━━━━<br/>{range}, {levels} levels"]
        F3["{Factor 3}<br/>━━━━━━━━━━<br/>{range}, {levels} levels"]
    end

    subgraph Strategy ["EXPLORATION STRATEGY"]
        direction TB
        STRAT["{Strategy Type}<br/>━━━━━━━━━━<br/>{e.g., Bayesian/Grid/OFAT}"]
    end

    subgraph Rounds ["RESULTS SO FAR"]
        direction TB
        R1["{Round 1}<br/>━━━━━━━━━━<br/>{n} runs, {finding}"]
        R2["{Round 2}<br/>━━━━━━━━━━<br/>{n} runs, {finding}"]
    end

    subgraph Next ["NEXT EXPERIMENTS"]
        direction TB
        NX1["{Next Experiment 1}<br/>━━━━━━━━━━<br/>{rationale}"]
        NX2["{Next Experiment 2}<br/>━━━━━━━━━━<br/>{rationale}"]
    end

    subgraph Stopping ["STOPPING CRITERIA"]
        direction TB
        STOP["{Stopping Rule}<br/>━━━━━━━━━━<br/>{threshold or condition}"]
    end

    OBJ["{Learning Objective}<br/>━━━━━━━━━━<br/>{key uncertainty to reduce}"]
    UNEXPLORED["{High-Value Unexplored Region}<br/>━━━━━━━━━━<br/>{what is missing and why it matters}"]

    OBJ --> F1
    OBJ --> F2
    OBJ --> F3
    F1 --> STRAT
    F2 --> STRAT
    F3 --> STRAT
    STRAT --> R1
    R1 --> R2
    R2 --> NX1
    R2 --> NX2
    NX1 --> STOP
    NX2 --> STOP
    UNEXPLORED -.->|missed| STRAT

    %% CLASS ASSIGNMENTS %%
    class F1,F2,F3 stateNode;
    class STRAT handler;
    class R1,R2 phase;
    class NX1,NX2 output;
    class OBJ cli;
    class UNEXPLORED gap;
    class STOP detector;

Color Legend:

Efficiency Assessment

Recommended Next Steps

---

## Pre-Diagram Checklist

Before creating the diagram, verify:

- [ ] LOADED `/mermaid` skill using the Skill tool
- [ ] Using ONLY classDef styles from the mermaid skill (no invented colors)
- [ ] Diagram will include a color legend table

---

## Related Skills

- `/make-experiment-diag` - Parent skill for lens selection
- `/mermaid` - MUST BE LOADED before creating diagram
- `/exp-lens-sensitivity-robustness` - For factor sensitivity analysis
- `/exp-lens-error-budget` - For cost and resource constraint analysis