Test Driven Development

Validation-First Development

Overview

Define success criteria first. Verify your validation catches problems. Then implement minimal solution.

Core principle: If you didn't verify your validation catches failures, you don't know if it validates the right thing.

Violating the letter of the rules is violating the spirit of the rules.

When to Use

Always:

• New features/experiments
• Bug fixes/anomaly investigations
• Refactoring/methodology improvements
• Behavior/hypothesis changes

Exceptions (ask your human partner or RD):

• Throwaway prototypes
• Generated code
• Configuration files
• Pure exploratory analysis (but then: throw away, restart with validation)

Thinking "skip validation-first just this once"? Stop. That's rationalization.

The Iron Law

NO IMPLEMENTATION WITHOUT DEFINED VALIDATION CRITERIA FIRST

Write code/run experiments before defining success criteria? Delete it. Start over.

No exceptions:

• Don't keep it as "reference"
• Don't "adapt" it while writing validation
• Don't look at it
• Delete means delete

Implement fresh from validation criteria. Period.

Red-Green-Refactor (Hypothesis-Test-Improve)

For Software: Classic TDD

RED    → Write failing test
GREEN  → Minimal code to pass
REFACTOR → Clean up, stay green

For Research: Validation-First

DEFINE   → State expected outcome and validation criteria
VERIFY   → Confirm validation would catch failure
EXECUTE  → Minimal experiment/analysis to produce result
VALIDATE → Check result against criteria
IMPROVE  → Refine methodology, stay valid

Software: Red-Green-Refactor

RED - Write Failing Test

Write one minimal test showing what should happen.

Good:

def test_retries_failed_operations_3_times():
    attempts = []
    def operation():
        attempts.append(1)
        if len(attempts) < 3:
            raise Exception('fail')
        return 'success'

    result = retry_operation(operation)

    assert result == 'success'
    assert len(attempts) == 3

Clear name, tests real behavior, one thing.

Bad:

def test_retry_works():
    mock = Mock(side_effect=[Exception(), Exception(), 'success'])
    retry_operation(mock)
    assert mock.call_count == 3

Vague name, tests mock not code.

Requirements:

• One behavior
• Clear name
• Real code (no mocks unless unavoidable)

Verify RED - Watch It Fail

MANDATORY. Never skip.

pytest tests/path/test.py::test_name -v

Confirm:

• Test fails (not errors)
• Failure message is expected
• Fails because feature missing (not typos)

Test passes? You're testing existing behavior. Fix test.

Test errors? Fix error, re-run until it fails correctly.

GREEN - Minimal Code

Write simplest code to pass the test. Nothing more.

Don't add features, refactor other code, or "improve" beyond the test.

Verify GREEN - Watch It Pass

MANDATORY.

pytest tests/path/test.py -v

Confirm:

• Test passes
• Other tests still pass
• Output pristine (no errors, warnings)

Test fails? Fix code, not test.

REFACTOR - Clean Up

After green only:

• Remove duplication
• Improve names
• Extract helpers

Keep tests green. Don't add behavior.

Research: Validation-First Development

DEFINE - State Expected Outcome

Before running any experiment or analysis, explicitly state:

Good:

## Expected Outcome
- Metric: Correlation between treatment and response
- Success: r > 0.3 with p < 0.05
- Failure: r < 0.1 or p > 0.1
- Anomaly: r > 0.9 (suspicious, check for confounds)

Bad:

## Expected Outcome
- We expect to see an effect
- Results should be significant

Requirements:

• Specific measurable criteria
• Explicit success threshold
• Explicit failure/null criteria
• What would indicate problems (anomalies, data quality issues)

VERIFY - Confirm Validation Works

MANDATORY. Never skip.

Before running your experiment, verify your validation criteria would catch problems:

1. What would a null result look like? (e.g., r ≈ 0, p > 0.5)
2. What would a confounded result look like? (e.g., perfect correlation due to batch effects)
3. What would bad data look like? (e.g., missing values, outliers, wrong format)

If you can't describe what failure looks like, your validation is worthless.

EXECUTE - Minimal Experiment/Analysis

Run the simplest version that tests your hypothesis. Nothing more.

Good:

# Minimal analysis
df = pd.read_csv("data.csv")
r, p = stats.pearsonr(df["treatment"], df["response"])
print(f"r={r:.3f}, p={p:.4f}")

Bad:

# Over-engineered before knowing if basic approach works
class CorrelationAnalyzer:
    def __init__(self, config, logger, cache, ...):
        # YAGNI

Don't add features, optimize, or expand scope beyond the immediate test.

VALIDATE - Check Against Criteria

MANDATORY.

Compare actual result to defined criteria:

• Does it meet success threshold?
• Does it fall in failure range?
• Are there any anomaly flags?

Result meets criteria? Proceed to improve.

Result in failure range? This is valid information. Document and report.

Result triggers anomalies? Investigate before trusting.

IMPROVE - Refine Methodology

Only after validation passes:

• Improve statistical power
• Add additional checks
• Clean up code
• Expand scope

Keep validation passing. Don't change hypothesis after seeing data.

Why Order Matters

"I'll add validation after to verify it works"

Validation added after passes immediately. Passing immediately proves nothing:

• Might validate wrong thing
• Might validate implementation artifacts, not real effects
• Might miss edge cases you forgot
• You never saw it catch a problem

Validation-first forces you to see it fail, proving it actually validates something.

"I already manually checked the results"

Manual checking is ad-hoc. You think you checked everything but:

• No record of what you checked
• Can't re-run when data changes
• Easy to forget checks under pressure
• "It looked right" ≠ systematic validation

Defined criteria are systematic. They check the same way every time.

"Deleting X hours of work is wasteful"

Sunk cost fallacy. The time is already gone. Your choice now:

• Delete and redo with validation-first (X more hours, high confidence)
• Keep it and add validation after (30 min, low confidence, likely problems)

The "waste" is keeping results you can't trust.

"This is exploratory, validation doesn't apply"

Exploratory is fine. But:

• Treat exploratory as throwaway
• When you find something interesting, STOP
• Define validation criteria for the finding
• Re-run with validation-first to confirm

"I explored, found X, and reported X" without validation is unreliable.

Common Rationalizations

Red Flags - STOP and Start Over

• Implementation before validation criteria defined
• Validation added after implementation
• Validation passes immediately without ever failing
• Can't explain what failure would look like
• Validation added "later"
• Rationalizing "just this once"
• "I already manually checked it"
• "Validation after achieves the same purpose"
• "Keep as reference" or "adapt existing code/results"
• "Already spent X hours, deleting is wasteful"
• "This is different because..."

All of these mean: Delete implementation. Start over with validation-first.

Example: Bug Fix (Software)

Bug: Empty email accepted

RED

def test_rejects_empty_email():
    result = submit_form({"email": ""})
    assert result["error"] == "Email required"

Verify RED

$ pytest test_form.py::test_rejects_empty_email -v
FAIL: expected 'Email required', got None

GREEN

def submit_form(data):
    if not data.get("email", "").strip():
        return {"error": "Email required"}
    # ...

Verify GREEN

$ pytest test_form.py -v
PASS

Example: Analysis (Research)

Question: Is gene X expression correlated with treatment response?

DEFINE

## Expected Outcome
- Metric: Spearman correlation between gene X expression and response score
- Success: rho > 0.3, p < 0.05, n > 30
- Failure: |rho| < 0.1 or p > 0.1
- Anomaly: rho > 0.9 (check for technical artifacts)

VERIFY

• Null result: rho ≈ 0 with high p-value (validates we can detect no effect)
• Bad data: Missing values in response score (validates we check data quality)
• Confound: Perfect correlation with batch (validates we check for artifacts)

EXECUTE

df = pd.read_csv("expression_data.csv")
rho, p = stats.spearmanr(df["geneX"], df["response"])
n = len(df)
print(f"rho={rho:.3f}, p={p:.4f}, n={n}")

VALIDATE

• Result: rho=0.42, p=0.003, n=45
• Meets success criteria? Yes (rho > 0.3, p < 0.05, n > 30)
• Anomaly flags? No (rho not suspiciously high)

IMPROVE

• Add confidence intervals
• Check for outliers
• Visualize relationship

Verification Checklist

Before marking work complete:

Software:

• Every new function/method has a test
• Watched each test fail before implementing
• Each test failed for expected reason (feature missing, not typo)
• Wrote minimal code to pass each test
• All tests pass
• Output pristine (no errors, warnings)

Research:

• Defined explicit success/failure criteria before executing
• Verified validation would catch known failure modes
• Executed minimal analysis/experiment
• Compared result to defined criteria
• Documented outcome (success, failure, or anomaly)
• Did NOT change criteria after seeing results

Can't check all boxes? You skipped validation-first. Start over.

When Stuck

Final Rule

Implementation/results → validation criteria existed and were verified first
Otherwise → not validation-first development

No exceptions without your human partner's (or RD's) permission.