Ml Benchmark Evaluation

ML Benchmark Evaluation — Rigorous Methodology

When to Use

• Claiming to beat published baselines on any ML benchmark
• Writing a methods paper that compares against prior work
• Any ML evaluation where the reported number matters for publication
• Auditing existing results for overfitting or data leakage

The Golden Rule

Your evaluation protocol must be AT LEAST as rigorous as the baseline you claim to beat. Ideally, stricter.

1. Train/Val/Test Split

The Wrong Way (common but weak):

train = data[:9000]   # 90%
test = data[9000:]    # 10% — used for BOTH model selection AND final metric
# Problem: model selection on test set inflates reported performance

The Right Way:

N_TRAIN, N_VAL, N_TEST = 8000, 1000, 1000
train = data[:N_TRAIN]                          # Gradient updates
val = data[N_TRAIN:N_TRAIN+N_VAL]               # Model selection (checkpoints)
test = data[N_TRAIN+N_VAL:N_TRAIN+N_VAL+N_TEST] # Final metric (ONCE)

# During training:
if epoch % 50 == 0:
    val_metric = evaluate(model, val_loader)  # NOT test_loader
    if val_metric < best_val_metric:
        save_checkpoint(model)

# After training (ONCE):
model = load_best_checkpoint()
final_metric = evaluate(model, test_loader)  # This is the reported number

When the Benchmark Doesn't Use Val Split:

Many benchmarks (PDEBench, some Kaggle, older CV benchmarks) don't use validation splits. When claiming to beat them:

1. Report results using their protocol for fair comparison
2. ALSO report results using proper val split for scientific rigor
3. Be transparent about both numbers in the paper

2. Verify Published Baselines (NEVER Trust Task Descriptions)

Published numbers can be wrong in secondary sources. Always verify from the original paper.

# Download and parse the original paper
wget -O paper.pdf "https://arxiv.org/pdf/XXXX.XXXXX"
npm i -g @llamaindex/liteparse
liteparse parse paper.pdf -o paper_parsed.md
grep "nRMSE\|accuracy\|F1" paper_parsed.md

Common discrepancies found in practice:

• Task specification says 5.9e-3 but paper says 9.7e-3 (wrong table, wrong metric)
• Baseline from a different parameter setting or model configuration
• RMSE vs nRMSE vs MSE confusion
• Different train/test split than claimed

3. Data Leak Detection Checklist

Run these 6 checks before reporting any result:

# Check 1: IC window preserved exactly (for time-series / PDE problems)
assert np.allclose(preds[:,:,:INIT_STEP], targets[:,:,:INIT_STEP], atol=1e-10)

# Check 2: Predictions differ from targets in predicted window
assert not np.allclose(preds[:,:,INIT_STEP:], targets[:,:,INIT_STEP:], atol=1e-6)

# Check 3: No duplicate samples between train and test
train_hashes = set(hash(x.tobytes()) for x in train_data)
test_hashes = set(hash(x.tobytes()) for x in test_data)
assert len(train_hashes & test_hashes) == 0

# Check 4: Test set never used for gradients
# Verify by code inspection: test_loader only in torch.no_grad() blocks

# Check 5: No NaN/Inf in predictions
assert not np.isnan(preds).any() and not np.isinf(preds).any()

# Check 6: Error distribution is realistic
# If min error is near-zero for many samples, suspicious
assert (per_sample_error < 1e-6).mean() < 0.01  # <1% near-perfect

4. Metric Computation (Get It Exactly Right)

Different benchmarks use different metrics. Verify the EXACT formula from the benchmark code.

# Example: PDEBench nRMSE
def calc_nrmse(preds, targets, init_step):
    """PDEBench nRMSE: per-timestep spatial RMSE, normalized, averaged."""
    p = preds[:,:,init_step:,:].permute(0,3,1,2)   # [N, C, X, T]
    tg = targets[:,:,init_step:,:].permute(0,3,1,2)
    err = torch.sqrt(torch.mean((p-tg)**2, dim=2))  # spatial RMSE: [N, C, T]
    nrm = torch.sqrt(torch.mean(tg**2, dim=2)) + 1e-20
    return torch.mean(err / nrm).item()

# ALWAYS cross-check against benchmark's own metrics code
# e.g., pdebench/models/metrics.py

CRITICAL: Metric Formula Discrepancies

The SAME metric name (e.g., "nRMSE") can have multiple valid definitions that give different numbers (up to 5-10% difference):

# Formula A: Per-timestep, then average (common in code implementations)
err_per_t = sqrt(mean_spatial((pred-target)^2))  # [N, C, T]
nrm_per_t = sqrt(mean_spatial(target^2))
nrmse_A = mean(err_per_t / nrm_per_t)  # average over N, C, T

# Formula B: Frobenius norm ratio per sample (canonical PDEBench definition)
nrmse_B = mean_over_N(||pred_i - target_i||_F / ||target_i||_F)

# Formula C: Global RMSE / global RMS
nrmse_C = sqrt(mean_all((pred-target)^2)) / sqrt(mean_all(target^2))

These are NOT equivalent. For a single benchmark comparison, the difference can be 1-10%. Always:

1. Read the benchmark's actual metrics.py code (not just the paper)
2. Compute your metric using the EXACT same formula
3. If in doubt, report BOTH formulas and show you beat under both
4. Document which formula you used in results.json

5. Multi-Seed Robustness

Single-seed results can be lucky. For strong claims:

seeds = [42, 123, 7, 2024, 31415]
results = []
for seed in seeds:
    torch.manual_seed(seed)
    model = train(seed)
    results.append(evaluate(model))

mean = np.mean(results)
std = np.std(results)
print(f"nRMSE = {mean:.4e} ± {std:.4e} (n={len(seeds)} seeds)")

Minimum for publication: 3 seeds for key results, report mean ± std.

Preserve the execution evidence

• Freeze the task/data hash, model revision, full treatment and decoding configuration, evaluator revision, environment and all random-seed policies before a measured run.
• Keep evaluator seeds and hidden tests outside the candidate's readable workspace. Retain the actual private seed ledger in protected evaluator storage and record its hash in the public run manifest. A hash without a retained ledger cannot reproduce the evaluation. Do not expose private seeds merely to make a run reproducible.
• Record every attempted sample, including invalid outputs, timeouts, judge failures and retries. Keep raw judge responses, subprocess exit status and stderr alongside normalized scores; never silently drop failures from the denominator.
• A successful notebook or tool invocation does not prove its nested test process passed. Check that subprocess's exit code and test summary. Report fake-provider, CPU, container and live-model checks separately; passing one does not establish the others.
• Verify shared environments and required imports before delegating dependent work. Pass the tested executable path and wait for setup's terminal result. A created virtual-environment directory alone is not a readiness check.
• Write run results atomically and validate manifest identity, expected samples and checksums before marking them complete. Resume only matching partial work; do not count an interrupted run as completed or mix different models into one frontier.

6. Honest Reporting Template

## Results

| Test | Our nRMSE | Published | Improvement | Seeds | Val Split |
|------|-----------|-----------|-------------|-------|-----------|
| A    | X.Xe-Y ± Z.Ze-Y | P.Pe-Q | N.N× | 3 | Yes (8K/1K/1K) |

### Comparison Fairness Notes:
- Our model uses [more modes / higher resolution / ...] than the baseline
- These confounding factors are documented in Table X
- Ablation study (Table Y) isolates the contribution of each change

### Limitations:
- [List every weakness honestly]

7. Physics-Informed Validation (for PDE/Scientific ML)

Beyond standard ML metrics, verify physical consistency:

Common Pitfalls