Validation Protocol
Overview
This document defines the complete validation protocol for materials simulations. Each stage has specific checks, criteria, and actions.
Stage 1: Pre-flight Checks
Run before simulation starts. Any BLOCKER prevents simulation launch.
1.1 Configuration Validation
| Check |
Criteria |
Severity |
| Required parameters present |
All mandatory fields exist in config |
BLOCKER |
| Parameter types correct |
Numeric fields are numbers, not strings |
BLOCKER |
| Parameter ranges valid |
Values within physical bounds |
BLOCKER |
| Units consistent |
All parameters use compatible units |
WARNING |
1.2 Stability Checks
| Check |
Criteria |
Severity |
| CFL condition |
dt <= dx / v_max for explicit schemes |
BLOCKER |
| Diffusion stability |
dt <= dx^2 / (2 * D * dim) |
BLOCKER |
| Courant number |
C = v * dt / dx < 1 for advection |
WARNING |
| Mesh resolution |
dx resolves smallest features |
WARNING |
1.3 Material Properties
| Check |
Criteria |
Severity |
| Source documented |
Material data has citation/source |
WARNING |
| Physical bounds |
Properties are physically reasonable |
BLOCKER |
| Temperature dependence |
If T-dependent, valid for T range |
WARNING |
| Consistency |
No contradictory property values |
BLOCKER |
1.4 System Resources
| Check |
Criteria |
Severity |
| Output directory exists |
Path is valid and accessible |
WARNING |
| Directory writable |
Write permissions confirmed |
BLOCKER |
| Sufficient disk space |
Free space > estimated output size |
BLOCKER |
| Memory estimate |
RAM sufficient for mesh size |
WARNING |
1.5 Pre-flight Decision Matrix
| Blockers |
Warnings |
Action |
| 0 |
0 |
PASS - Proceed immediately |
| 0 |
1+ |
WARN - Review and document |
| 1+ |
Any |
BLOCK - Do not proceed |
Stage 2: Runtime Checks
Monitor during simulation execution. Alert thresholds are configurable.
2.1 Convergence Monitoring
| Metric |
Alert Condition |
Likely Cause |
| Residual growth |
r[n] / r[n-1] > threshold |
Instability, bad IC |
| Residual stagnation |
r[n] ≈ r[n-k] for many k |
Solver stuck |
| Iteration count |
Near max iterations |
Ill-conditioned system |
Default Thresholds:
- Residual growth threshold: 10x (single step)
- Cumulative growth threshold: 1000x (over run)
- Stagnation window: 100 iterations
2.2 Time Step Monitoring
| Metric |
Alert Condition |
Likely Cause |
| dt reduction |
dt_max / dt_min > threshold |
Stability issues |
| dt collapse |
dt < dt_min_allowed |
Imminent failure |
| dt oscillation |
Repeated up/down cycles |
Instability boundary |
Default Thresholds:
- dt reduction threshold: 100x
- dt minimum: 1e-12 (problem-dependent)
2.3 Conservation Monitoring
| Quantity |
Alert Condition |
Tolerance |
| Mass |
` |
M(t) - M(0) |
| Energy |
E(t) > E(0) (if dissipative) |
1e-6 |
| Momentum |
` |
P(t) - P(0) |
2.4 Runtime Alert Severity
| Severity |
Action |
| INFO |
Log only, no action needed |
| WARNING |
Continue but monitor closely |
| CRITICAL |
Consider stopping simulation |
| FATAL |
Stop immediately, diagnose |
Stage 3: Post-flight Checks
Run after simulation completes. Failed checks invalidate results.
3.1 Conservation Validation
| Check |
Criteria |
Action if Failed |
| Mass conserved |
Drift < tolerance |
Results invalid |
| Energy behavior |
Monotonic if variational |
Review physics |
| Momentum conserved |
Drift < tolerance |
Results invalid |
3.2 Field Validation
| Check |
Criteria |
Action if Failed |
| No NaN values |
count(NaN) == 0 |
Results invalid |
| No Inf values |
count(Inf) == 0 |
Results invalid |
| Bounds respected |
field_min >= bound_min |
Review BC/IC |
| Physical range |
Values physically meaningful |
Review model |
Common Field Bounds:
- Phase field:
0 <= phi <= 1
- Concentration:
0 <= c <= c_max
- Temperature:
T > 0 (absolute)
- Density:
rho > 0
3.3 Statistical Validation
| Check |
Criteria |
Notes |
| Mean reasonable |
Within expected range |
Problem-dependent |
| Variance stable |
Not exploding over time |
Check late times |
| Extremes bounded |
No outliers beyond physical |
Check mesh refinement |
3.4 Confidence Score Calculation
confidence = (passed_checks) / (total_checks)
Interpretation:
- 1.0: All checks passed, high confidence
- 0.75-0.99: Minor issues, review before use
- 0.5-0.74: Significant issues, use with caution
- < 0.5: Major problems, do not use results
Validation Documentation
What to Record
For each simulation, document:
- Pre-flight status and any accepted warnings
- Runtime alerts and actions taken
- Post-flight validation results
- Confidence score
- Any manual review notes
Traceability
Link validation records to:
- Simulation configuration (version-controlled)
- Software version and commit hash
- Hardware/environment details
- Date and operator
Quick Reference Checklist
Before Running
During Run
After Run
tools-only