Benchmark And MMS Planner
Goal
Design a verification and validation plan before trusting simulation results. The skill helps agents choose manufactured solutions, benchmark cases, refinement protocols, uncertainty checks, and pass/fail criteria.
Requirements
- Python 3.10+
- No external dependencies
- Works on Linux, macOS, and Windows
Inputs to Gather
| Input |
Description |
Example |
| PDE or model class |
Governing family |
diffusion, elasticity, phase-field |
| Quantity of interest |
Metric to validate |
interface velocity, L2 temperature error |
| Dimension |
1, 2, or 3 |
2 |
| Expected order |
Formal discretization order |
2 |
| Reference availability |
Analytic, benchmark, or none |
analytic |
| Risk level |
Cost or consequence of wrong result |
high |
Decision Guidance
- Use MMS when code correctness is uncertain and an analytic solution can be injected.
- Use canonical benchmarks when physical model validation matters more than code verification.
- Use grid/time refinement whenever the result is used for a claim, design decision, or comparison.
- Use uncertainty propagation when inputs are calibrated, noisy, or experimentally measured.
Script Outputs
scripts/benchmark_mms_planner.py emits inputs and results with:
verification_strategymms_planbenchmark_casesrefinement_protocolacceptance_criteriawarnings
Workflow
- Collect the governing model, quantity of interest, and risk level.
- Run
benchmark_mms_planner.py --json.
- Treat warnings as blockers for high-risk claims.
- Convert the returned protocol into tests, simulation runs, or review checklist items.
python3 skills/verification-validation/benchmark-and-mms-planner/scripts/benchmark_mms_planner.py \
--model diffusion \
--quantity "L2 error in temperature" \
--dimension 2 \
--expected-order 2 \
--reference analytic \
--risk high \
--json
Error Handling
- If the dimension or expected order is invalid, stop and correct the model description.
- If no reference exists, use conservation and convergence checks but do not call the result validated.
Limitations
This skill plans verification work; it does not run the solver or prove that a physical model is appropriate for an experiment.
Security
- Inputs are scalar strings and finite numeric values only.
- The script does not execute external solvers.
- File writes are not performed.
- The skill uses
Bash only to run its bundled script.
References
- See
references/vv_patterns.md for MMS, benchmark, and uncertainty planning notes.
Version History
- 1.0.0: Initial benchmark and MMS planning skill.
1---2name: benchmark-and-mms-planner3description: Plan verification and validation campaigns for simulation codes using manufactured solutions, canonical benchmark problems, grid/time refinement, uncertainty propagation, and pass/fail acceptance criteria. Use when an agent needs to prove a solver, model, or result is trustworthy rather than only plausible.4---56# Benchmark And MMS Planner78## Goal910Design a verification and validation plan before trusting simulation results. The skill helps agents choose manufactured solutions, benchmark cases, refinement protocols, uncertainty checks, and pass/fail criteria.1112## Requirements1314- Python 3.10+15- No external dependencies16- Works on Linux, macOS, and Windows1718## Inputs to Gather1920| Input | Description | Example |21|-------|-------------|---------|22| PDE or model class | Governing family | `diffusion`, `elasticity`, `phase-field` |23| Quantity of interest | Metric to validate | `interface velocity`, `L2 temperature error` |24| Dimension | 1, 2, or 3 | `2` |25| Expected order | Formal discretization order | `2` |26| Reference availability | Analytic, benchmark, or none | `analytic` |27| Risk level | Cost or consequence of wrong result | `high` |2829## Decision Guidance3031- Use **MMS** when code correctness is uncertain and an analytic solution can be injected.32- Use **canonical benchmarks** when physical model validation matters more than code verification.33- Use **grid/time refinement** whenever the result is used for a claim, design decision, or comparison.34- Use **uncertainty propagation** when inputs are calibrated, noisy, or experimentally measured.3536## Script Outputs3738`scripts/benchmark_mms_planner.py` emits `inputs` and `results` with:3940- `verification_strategy`41- `mms_plan`42- `benchmark_cases`43- `refinement_protocol`44- `acceptance_criteria`45- `warnings`4647## Workflow48491. Collect the governing model, quantity of interest, and risk level.502. Run `benchmark_mms_planner.py --json`.513. Treat warnings as blockers for high-risk claims.524. Convert the returned protocol into tests, simulation runs, or review checklist items.5354```bash55python3 skills/verification-validation/benchmark-and-mms-planner/scripts/benchmark_mms_planner.py \56 --model diffusion \57 --quantity "L2 error in temperature" \58 --dimension 2 \59 --expected-order 2 \60 --reference analytic \61 --risk high \62 --json63```6465## Error Handling6667- If the dimension or expected order is invalid, stop and correct the model description.68- If no reference exists, use conservation and convergence checks but do not call the result validated.6970## Limitations7172This skill plans verification work; it does not run the solver or prove that a physical model is appropriate for an experiment.7374## Security7576- Inputs are scalar strings and finite numeric values only.77- The script does not execute external solvers.78- File writes are not performed.79- The skill uses `Bash` only to run its bundled script.8081## References8283- See `references/vv_patterns.md` for MMS, benchmark, and uncertainty planning notes.8485## Version History8687- 1.0.0: Initial benchmark and MMS planning skill.
zuoguyoupan2023