Simulation Validator
Goal
Provide a three-stage validation protocol: pre-flight checks, runtime monitoring, and post-flight validation for materials simulations.
Requirements
- Python 3.10+
- No external dependencies (uses Python standard library only)
- Works on Linux, macOS, and Windows
Inputs to Gather
Before running validation scripts, collect from the user:
| Input |
Description |
Example |
| Config file |
Simulation configuration (JSON/YAML) |
simulation.json |
| Log file |
Runtime output log |
simulation.log |
| Metrics file |
Post-run metrics (JSON) |
results.json |
| Required params |
Parameters that must exist |
dt,dx,kappa |
| Valid ranges |
Parameter bounds |
dt:1e-6:1e-2 |
Decision Guidance
When to Run Each Stage
Is simulation about to start?
├── YES → Run Stage 1: preflight_checker.py
│ └── BLOCK status? → Fix issues, do NOT run simulation
│ └── WARN status? → Review warnings, document if accepted
│ └── PASS status? → Proceed to run simulation
│
Is simulation running?
├── YES → Run Stage 2: runtime_monitor.py (periodically)
│ └── Alerts? → Consider stopping, check parameters
│
Has simulation finished?
├── YES → Run Stage 3: result_validator.py
│ └── Failed checks? → Do NOT use results
│ → Run failure_diagnoser.py
│ └── All passed? → Results are valid
Choosing Validation Thresholds
| Metric |
Conservative |
Standard |
Relaxed |
| Mass tolerance |
1e-6 |
1e-3 |
1e-2 |
| Residual growth |
2x |
10x |
100x |
| dt reduction |
10x |
100x |
1000x |
Script Outputs (JSON Fields)
| Script |
Output Fields |
scripts/preflight_checker.py |
report.status, report.blockers, report.warnings |
scripts/runtime_monitor.py |
alerts, residual_stats, dt_stats |
scripts/result_validator.py |
checks, confidence_score, failed_checks |
scripts/failure_diagnoser.py |
probable_causes, recommended_fixes |
Three-Stage Validation Protocol
Stage 1: Pre-flight (Before Simulation)
- Run
scripts/preflight_checker.py --config simulation.json
- BLOCK status: Stop immediately, fix all blocker issues
- WARN status: Review warnings, document accepted risks
- PASS status: Proceed to simulation
python3 scripts/preflight_checker.py \
--config simulation.json \
--required dt,dx,kappa \
--ranges "dt:1e-6:1e-2,dx:1e-4:1e-1" \
--min-free-gb 1.0 \
--json
Stage 2: Runtime (During Simulation)
- Run
scripts/runtime_monitor.py --log simulation.log periodically
- Configure alert thresholds based on problem type
- Stop simulation if critical alerts appear
python3 scripts/runtime_monitor.py \
--log simulation.log \
--residual-growth 10.0 \
--dt-drop 100.0 \
--json
Stage 3: Post-flight (After Simulation)
- Run
scripts/result_validator.py --metrics results.json
- All checks PASS: Results are valid for analysis
- Any check FAIL: Do NOT use results, diagnose failure
python3 scripts/result_validator.py \
--metrics results.json \
--bound-min 0.0 \
--bound-max 1.0 \
--mass-tol 1e-3 \
--json
Failure Diagnosis
When validation fails:
python3 scripts/failure_diagnoser.py --log simulation.log --json
Conversational Workflow Example
User: My phase field simulation crashed after 1000 steps. Can you help me figure out why?
Agent workflow:
- First, check the log for obvious errors:
python3 scripts/failure_diagnoser.py --log simulation.log --json
- If diagnosis suggests numerical blow-up, check runtime stats:
python3 scripts/runtime_monitor.py --log simulation.log --json
- Recommend fixes based on findings:
- If residual grew rapidly → reduce time step
- If dt collapsed → check stability conditions
- If NaN detected → check initial conditions
Error Handling
| Error |
Cause |
Resolution |
Config not found |
File path invalid |
Verify config path exists |
Non-numeric value |
Parameter is not a number |
Fix config file format |
out of range |
Parameter outside bounds |
Adjust parameter or bounds |
Output directory not writable |
Permission issue |
Check directory permissions |
Insufficient disk space |
Disk nearly full |
Free up space or reduce output |
Interpretation Guidance
Status Meanings
| Status |
Meaning |
Action |
| PASS |
All checks passed |
Proceed with confidence |
| WARN |
Non-critical issues found |
Review and document |
| BLOCK |
Critical issues found |
Must fix before proceeding |
Confidence Score Interpretation
| Score |
Meaning |
| 1.0 |
All validation checks passed |
| 0.75+ |
Most checks passed, minor issues |
| 0.5-0.75 |
Significant issues, review carefully |
| < 0.5 |
Major problems, do not trust results |
Common Failure Patterns
| Pattern in Log |
Likely Cause |
Recommended Fix |
| NaN, Inf, overflow |
Numerical instability |
Reduce dt, increase damping |
| max iterations, did not converge |
Solver failure |
Tune preconditioner, tolerances |
| out of memory |
Memory exhaustion |
Reduce mesh, enable out-of-core |
| dt reduced |
Adaptive stepping triggered |
May be okay if controlled |
Security
Input Validation
- Config file paths are validated for existence before parsing; non-existent paths produce clear errors
--required parameter names are validated against a safe-character allowlist
--ranges entries are parsed as name:min:max with finite numeric bounds enforced
--min-free-gb is validated as a finite positive number
--residual-growth and --dt-drop thresholds are validated as finite positive numbers
--bound-min, --bound-max, and --mass-tol are validated as finite numbers with bound-max > bound-min
File Access
preflight_checker.py reads a single user-specified config file (JSON/YAML) and checks disk space on the output directory
runtime_monitor.py reads a single log file specified by --log; log files are size-limited (500 MB max) before parsing
result_validator.py reads a single metrics file (JSON) specified by --metrics
failure_diagnoser.py reads a single log file specified by --log
- No scripts write to the filesystem; all output goes to stdout
Tool Restrictions
- Read: Used to inspect script source, references, config files, and simulation logs
- Bash: Used to execute the four Python validation scripts (
preflight_checker.py, runtime_monitor.py, result_validator.py, failure_diagnoser.py) with explicit argument lists
- Write: Used to save validation reports; writes are scoped to the user's working directory
- Grep/Glob: Used to locate log files, config files, and search references
Safety Measures
- No
eval(), exec(), or dynamic code generation
- All subprocess calls use explicit argument lists (no
shell=True)
- Log parsing uses pre-compiled regex patterns; user-supplied patterns are not accepted (patterns are hardcoded)
- Phase names and diagnostic strings extracted from logs are sanitized (truncated, control characters stripped) before inclusion in output
Limitations
- Not a real-time monitor: Scripts analyze logs after-the-fact
- Regex-based: Log parsing depends on pattern matching; may miss unusual formats
- No automatic fixes: Scripts diagnose but don't modify simulations
References
references/validation_protocol.md - Detailed checklist and criteria
references/log_patterns.md - Common failure signatures and regex patterns
Version History
- v1.1.0 (2024-12-24): Enhanced documentation, decision guidance, Windows compatibility
- v1.0.0: Initial release with 4 validation scripts
1---2name: simulation-validator3description: >4 Validate simulations across three stages — run pre-flight checks on5 configuration files (parameter ranges, required fields, disk space),6 monitor runtime logs for residual growth, NaN/Inf, and adaptive dt7 collapse, and perform post-flight validation of results (physical bounds,8 mass/energy conservation, convergence). Diagnose failed simulations with9 probable-cause analysis and recommended fixes. Use when preparing to10 launch a simulation, checking whether a running job is healthy, verifying11 that finished results are trustworthy, or debugging a crash or blow-up,12 even if the user only says "my simulation crashed" or "can I trust13 these results."14allowed-tools: Read, Bash, Write, Grep, Glob15metadata:16 source: https://skillmd.com/skills/zuoguyoupan2023/simulation-validator17 author: HeshamFS18 version: "1.1.0"19 security_tier: high20 security_reviewed: true21 tested_with:22 - claude-code23 - gemini-cli24 - vs-code-copilot25 eval_cases: 526 last_reviewed: "2026-03-26"27---2829# Simulation Validator3031## Goal3233Provide a three-stage validation protocol: pre-flight checks, runtime monitoring, and post-flight validation for materials simulations.3435## Requirements3637- Python 3.10+38- No external dependencies (uses Python standard library only)39- Works on Linux, macOS, and Windows4041## Inputs to Gather4243Before running validation scripts, collect from the user:4445| Input | Description | Example |46|-------|-------------|---------|47| Config file | Simulation configuration (JSON/YAML) | `simulation.json` |48| Log file | Runtime output log | `simulation.log` |49| Metrics file | Post-run metrics (JSON) | `results.json` |50| Required params | Parameters that must exist | `dt,dx,kappa` |51| Valid ranges | Parameter bounds | `dt:1e-6:1e-2` |5253## Decision Guidance5455### When to Run Each Stage5657```58Is simulation about to start?59├── YES → Run Stage 1: preflight_checker.py60│ └── BLOCK status? → Fix issues, do NOT run simulation61│ └── WARN status? → Review warnings, document if accepted62│ └── PASS status? → Proceed to run simulation63│64Is simulation running?65├── YES → Run Stage 2: runtime_monitor.py (periodically)66│ └── Alerts? → Consider stopping, check parameters67│68Has simulation finished?69├── YES → Run Stage 3: result_validator.py70│ └── Failed checks? → Do NOT use results71│ → Run failure_diagnoser.py72│ └── All passed? → Results are valid73```7475### Choosing Validation Thresholds7677| Metric | Conservative | Standard | Relaxed |78|--------|--------------|----------|---------|79| Mass tolerance | 1e-6 | 1e-3 | 1e-2 |80| Residual growth | 2x | 10x | 100x |81| dt reduction | 10x | 100x | 1000x |8283## Script Outputs (JSON Fields)8485| Script | Output Fields |86|--------|---------------|87| `scripts/preflight_checker.py` | `report.status`, `report.blockers`, `report.warnings` |88| `scripts/runtime_monitor.py` | `alerts`, `residual_stats`, `dt_stats` |89| `scripts/result_validator.py` | `checks`, `confidence_score`, `failed_checks` |90| `scripts/failure_diagnoser.py` | `probable_causes`, `recommended_fixes` |9192## Three-Stage Validation Protocol9394### Stage 1: Pre-flight (Before Simulation)95961. Run `scripts/preflight_checker.py --config simulation.json`972. **BLOCK status**: Stop immediately, fix all blocker issues983. **WARN status**: Review warnings, document accepted risks994. **PASS status**: Proceed to simulation100101```bash102python3 scripts/preflight_checker.py \103 --config simulation.json \104 --required dt,dx,kappa \105 --ranges "dt:1e-6:1e-2,dx:1e-4:1e-1" \106 --min-free-gb 1.0 \107 --json108```109110### Stage 2: Runtime (During Simulation)1111121. Run `scripts/runtime_monitor.py --log simulation.log` periodically1132. Configure alert thresholds based on problem type1143. Stop simulation if critical alerts appear115116```bash117python3 scripts/runtime_monitor.py \118 --log simulation.log \119 --residual-growth 10.0 \120 --dt-drop 100.0 \121 --json122```123124### Stage 3: Post-flight (After Simulation)1251261. Run `scripts/result_validator.py --metrics results.json`1272. **All checks PASS**: Results are valid for analysis1283. **Any check FAIL**: Do NOT use results, diagnose failure129130```bash131python3 scripts/result_validator.py \132 --metrics results.json \133 --bound-min 0.0 \134 --bound-max 1.0 \135 --mass-tol 1e-3 \136 --json137```138139### Failure Diagnosis140141When validation fails:142143```bash144python3 scripts/failure_diagnoser.py --log simulation.log --json145```146147## Conversational Workflow Example148149**User**: My phase field simulation crashed after 1000 steps. Can you help me figure out why?150151**Agent workflow**:1521. First, check the log for obvious errors:153 ```bash154 python3 scripts/failure_diagnoser.py --log simulation.log --json155 ```1562. If diagnosis suggests numerical blow-up, check runtime stats:157 ```bash158 python3 scripts/runtime_monitor.py --log simulation.log --json159 ```1603. Recommend fixes based on findings:161 - If residual grew rapidly → reduce time step162 - If dt collapsed → check stability conditions163 - If NaN detected → check initial conditions164165## Error Handling166167| Error | Cause | Resolution |168|-------|-------|------------|169| `Config not found` | File path invalid | Verify config path exists |170| `Non-numeric value` | Parameter is not a number | Fix config file format |171| `out of range` | Parameter outside bounds | Adjust parameter or bounds |172| `Output directory not writable` | Permission issue | Check directory permissions |173| `Insufficient disk space` | Disk nearly full | Free up space or reduce output |174175## Interpretation Guidance176177### Status Meanings178179| Status | Meaning | Action |180|--------|---------|--------|181| PASS | All checks passed | Proceed with confidence |182| WARN | Non-critical issues found | Review and document |183| BLOCK | Critical issues found | Must fix before proceeding |184185### Confidence Score Interpretation186187| Score | Meaning |188|-------|---------|189| 1.0 | All validation checks passed |190| 0.75+ | Most checks passed, minor issues |191| 0.5-0.75 | Significant issues, review carefully |192| < 0.5 | Major problems, do not trust results |193194### Common Failure Patterns195196| Pattern in Log | Likely Cause | Recommended Fix |197|----------------|--------------|-----------------|198| NaN, Inf, overflow | Numerical instability | Reduce dt, increase damping |199| max iterations, did not converge | Solver failure | Tune preconditioner, tolerances |200| out of memory | Memory exhaustion | Reduce mesh, enable out-of-core |201| dt reduced | Adaptive stepping triggered | May be okay if controlled |202203## Security204205### Input Validation206- Config file paths are validated for existence before parsing; non-existent paths produce clear errors207- `--required` parameter names are validated against a safe-character allowlist208- `--ranges` entries are parsed as `name:min:max` with finite numeric bounds enforced209- `--min-free-gb` is validated as a finite positive number210- `--residual-growth` and `--dt-drop` thresholds are validated as finite positive numbers211- `--bound-min`, `--bound-max`, and `--mass-tol` are validated as finite numbers with `bound-max > bound-min`212213### File Access214- `preflight_checker.py` reads a single user-specified config file (JSON/YAML) and checks disk space on the output directory215- `runtime_monitor.py` reads a single log file specified by `--log`; log files are size-limited (500 MB max) before parsing216- `result_validator.py` reads a single metrics file (JSON) specified by `--metrics`217- `failure_diagnoser.py` reads a single log file specified by `--log`218- No scripts write to the filesystem; all output goes to stdout219220### Tool Restrictions221- **Read**: Used to inspect script source, references, config files, and simulation logs222- **Bash**: Used to execute the four Python validation scripts (`preflight_checker.py`, `runtime_monitor.py`, `result_validator.py`, `failure_diagnoser.py`) with explicit argument lists223- **Write**: Used to save validation reports; writes are scoped to the user's working directory224- **Grep/Glob**: Used to locate log files, config files, and search references225226### Safety Measures227- No `eval()`, `exec()`, or dynamic code generation228- All subprocess calls use explicit argument lists (no `shell=True`)229- Log parsing uses pre-compiled regex patterns; user-supplied patterns are not accepted (patterns are hardcoded)230- Phase names and diagnostic strings extracted from logs are sanitized (truncated, control characters stripped) before inclusion in output231232## Limitations233234- **Not a real-time monitor**: Scripts analyze logs after-the-fact235- **Regex-based**: Log parsing depends on pattern matching; may miss unusual formats236- **No automatic fixes**: Scripts diagnose but don't modify simulations237238## References239240- `references/validation_protocol.md` - Detailed checklist and criteria241- `references/log_patterns.md` - Common failure signatures and regex patterns242243## Version History244245- **v1.1.0** (2024-12-24): Enhanced documentation, decision guidance, Windows compatibility246- **v1.0.0**: Initial release with 4 validation scripts