Data & Model Evaluation (agsy-data-and-model-evaluation)
A model is only as credible as its evaluation. AgSy reviewers are systems-modelling experts: they
want to see the model tested against independent data, its sensitivity and uncertainty
characterized, and the trade-offs the system exhibits — not a single tuned run presented as truth.
Model description and choice live in agsy-systems-framing-and-modeling; this skill covers testing and
reporting.
When to trigger
- Reporting how well the model reproduces observations
- Running sensitivity / uncertainty analysis
- Building scenario comparisons and trade-off analyses
- A reviewer asked for validation, sensitivity, uncertainty, or alternative scenarios
Evaluation norms AgSy expects
- Independent evaluation. Compare observed vs. simulated on data not used for calibration.
Report standard fit statistics — RMSE, RRMSE, bias/ME, modelling efficiency (NSE), index of
agreement (d), R² — and show the 1:1 plot. State what "good enough" means for the decision.
- Sensitivity analysis. Identify the parameters/inputs that drive outputs (local one-at-a-time
and, where feasible, global methods — Morris, Sobol). Report which assumptions matter most.
- Uncertainty. Propagate uncertainty from inputs, parameters, and structure to the outputs that
carry the conclusions. Present ranges/intervals, not point estimates dressed as certainty.
- Trade-off & scenario analysis. This is where AgSy papers earn their place: compare scenarios or
management/design options on multiple objectives (yield, profit, environment, risk) and show the
trade-offs and synergies — Pareto fronts, trade-off curves, multi-indicator profiles.
- Scaling. When you aggregate from field to farm to region, state how, and check that aggregation
does not hide compensating errors.
Stochastic & data-driven components
- For Monte Carlo, ABM, or stochastic weather/price generators: set and report seeds, run enough
replicates, and report the distribution, not one realization.
- For surrogate/emulator or ML components: validate against the full model and report where they fail.
Reproducibility while you work (not at the end)
- One master workflow regenerates every figure/table from inputs + model runs.
- Pin model version, parameter sets, and software versions; record calibration vs. evaluation splits.
- Keep table/figure numbers matched to script/model outputs — the data/code/model are deposited (see
agsy-reproducibility-and-data-policy).
Anti-patterns
- Calibrating and "validating" on the same data; reporting fit only on the training period
- A single tuned run with no sensitivity or uncertainty analysis
- Reporting R² alone (insensitive to bias) — pair it with RMSE/NSE/bias
- Scenario tables with no trade-offs surfaced (just "scenario B is best")
- Aggregating across scales without checking for compensating errors
Evaluation completeness rubric (what an AgSy referee checks)
A systems referee scores the model section on five axes (strong → weak, a weak answer often draws a
major revision): independence (held-out sites/years → "validated" on calibration data); fit (RMSE+NSE+
bias+1:1 plot → R² only); sensitivity (global Morris/Sobol → one-at-a-time); uncertainty (propagated
intervals → a discussion mention); trade-offs (Pareto front → a "best scenario" sentence).
Worked micro-example (illustrative numbers)
A bioeconomic whole-farm model evaluates climate-adaptation options for a mixed crop–livestock farm.
Numbers are illustrative — not real results.
- Evaluation. Calibrated on 2008–2015; held-out 2016–2020 gives margin RMSE = €185/ha, NSE = 0.74,
bias = −€20/ha with the 1:1 plot; "good enough" is RRMSE < 15%, and the negative bias is flagged.
- Sensitivity. Sobol indices: rainfall (0.41) and price (0.27) dominate, so results are reported on
a rainfall/price grid, not a single forecast.
- Trade-off. Option B raises margin +8% but N surplus +14%; D loses 3% margin but cuts N surplus
22%. The Pareto front is plotted; the recommendation names the trade-off, not a single winner.
Referee pushback → the AgSy-specific fix
- "Not validated against independent data." → Hold out sites/years, report the full fit panel on the
held-out set, and state the decision-relevant tolerance.
- "Trade-offs across objectives not quantified." → Replace "best scenario" with a multi-objective
table or Pareto front showing what each option gains and gives up.
Calibration anchors (hedged where policy is volatile)
- Fit-statistic conventions (RMSE, RRMSE, NSE, bias, d) are community norms, not a fixed journal rule.
- "Independent evaluation" means data withheld from calibration; the exact bar is judged by referees.
- Whether a data/code/model deposit is required versus encouraged is an Elsevier research-data question
— confirm against the journal's current author guidelines.
Output format
【Evaluation data】independent of calibration? [Y/N]
【Fit statistics】RMSE / NSE / bias / d / R² + 1:1 plot
【Sensitivity】which inputs/parameters dominate
【Uncertainty】propagated to the conclusion-bearing outputs? [Y/N]
【Trade-offs】scenarios compared on multiple objectives (Pareto/curve)
【Reproducible】master workflow + pinned model/version + seeds? [Y/N]
【Next】agsy-figures-and-tables
Supplementary resources
Source: brycewang-stanford/Awesome-Journal-Skills → Agricultural-Systems-Skills/skills/agsy-data-and-model-evaluation/SKILL.md
1---2name: agsy-data-and-model-evaluation3description: Use when evaluating the model and analyzing results for an Agricultural Systems (AgSy) manuscript so it survives expert systems review — independent model evaluation (observed vs. simulated, fit statistics), sensitivity and uncertainty analysis, and trade-off / scenario analysis across the system. Guides evaluation norms; it does not fabricate results or run the model.4---5
6
7# Data & Model Evaluation (agsy-data-and-model-evaluation)
8
9A model is only as credible as its **evaluation**. AgSy reviewers are systems-modelling experts: they
10want to see the model tested against **independent** data, its **sensitivity and uncertainty**
11characterized, and the **trade-offs** the system exhibits — not a single tuned run presented as truth.
12Model description and choice live in `agsy-systems-framing-and-modeling`; this skill covers testing and
13reporting.
14
15## When to trigger
16
17- Reporting how well the model reproduces observations
18- Running sensitivity / uncertainty analysis
19- Building scenario comparisons and trade-off analyses
20- A reviewer asked for validation, sensitivity, uncertainty, or alternative scenarios
21
22## Evaluation norms AgSy expects
23
241. **Independent evaluation.** Compare **observed vs. simulated** on data **not used for calibration**.
25 Report standard fit statistics — **RMSE, RRMSE, bias/ME, modelling efficiency (NSE), index of
26 agreement (d), R²** — and show the 1:1 plot. State what "good enough" means for the decision.
272. **Sensitivity analysis.** Identify the parameters/inputs that drive outputs (local one-at-a-time
28 and, where feasible, global methods — **Morris, Sobol**). Report which assumptions matter most.
293. **Uncertainty.** Propagate uncertainty from inputs, parameters, and structure to the outputs that
30 carry the conclusions. Present ranges/intervals, not point estimates dressed as certainty.
314. **Trade-off & scenario analysis.** This is where AgSy papers earn their place: compare scenarios or
32 management/design options on **multiple objectives** (yield, profit, environment, risk) and show the
33 **trade-offs and synergies** — Pareto fronts, trade-off curves, multi-indicator profiles.
345. **Scaling.** When you aggregate from field to farm to region, state how, and check that aggregation
35 does not hide compensating errors.
36
37## Stochastic & data-driven components
38- For Monte Carlo, ABM, or stochastic weather/price generators: **set and report seeds**, run enough
39 replicates, and report the distribution, not one realization.
40- For surrogate/emulator or ML components: validate against the full model and report where they fail.
41
42## Reproducibility while you work (not at the end)
43- One **master workflow** regenerates every figure/table from inputs + model runs.
44- Pin model version, parameter sets, and software versions; record calibration vs. evaluation splits.
45- Keep table/figure numbers matched to script/model outputs — the data/code/model are deposited (see
46 `agsy-reproducibility-and-data-policy`).
47
48## Anti-patterns
49
50- Calibrating and "validating" on the **same** data; reporting fit only on the training period
51- A single tuned run with no sensitivity or uncertainty analysis
52- Reporting R² alone (insensitive to bias) — pair it with RMSE/NSE/bias
53- Scenario tables with no trade-offs surfaced (just "scenario B is best")
54- Aggregating across scales without checking for compensating errors
55
56## Evaluation completeness rubric (what an AgSy referee checks)
57
58A systems referee scores the model section on five axes (**strong → weak**, a weak answer often draws a
59major revision): independence (held-out sites/years → "validated" on calibration data); fit (RMSE+NSE+
60bias+1:1 plot → R² only); sensitivity (global Morris/Sobol → one-at-a-time); uncertainty (propagated
61intervals → a discussion mention); trade-offs (Pareto front → a "best scenario" sentence).
62
63## Worked micro-example (illustrative numbers)
64
65A bioeconomic whole-farm model evaluates climate-adaptation options for a mixed crop–livestock farm.
66Numbers are **illustrative** — not real results.
67
68- **Evaluation.** Calibrated on 2008–2015; held-out 2016–2020 gives margin RMSE = €185/ha, NSE = 0.74,
69 bias = −€20/ha with the 1:1 plot; "good enough" is RRMSE < 15%, and the negative bias is flagged.
70- **Sensitivity.** Sobol indices: rainfall (0.41) and price (0.27) dominate, so results are reported on
71 a rainfall/price grid, not a single forecast.
72- **Trade-off.** Option B raises margin +8% but N surplus +14%; D loses 3% margin but cuts N surplus
73 22%. The Pareto front is plotted; the recommendation names the trade-off, not a single winner.
74
75## Referee pushback → the AgSy-specific fix
76
77- *"Not validated against independent data."* → Hold out sites/years, report the full fit panel on the
78 held-out set, and state the decision-relevant tolerance.
79- *"Trade-offs across objectives not quantified."* → Replace "best scenario" with a multi-objective
80 table or Pareto front showing what each option gains and gives up.
81
82## Calibration anchors (hedged where policy is volatile)
83
84- Fit-statistic conventions (RMSE, RRMSE, NSE, bias, d) are community norms, not a fixed journal rule.
85- "Independent evaluation" means data withheld from calibration; the exact bar is judged by referees.
86- Whether a data/code/model deposit is required versus encouraged is an Elsevier research-data question
87 — confirm against the journal's current author guidelines.
88
89## Output format
90
91```
92【Evaluation data】independent of calibration? [Y/N]
93【Fit statistics】RMSE / NSE / bias / d / R² + 1:1 plot
94【Sensitivity】which inputs/parameters dominate
95【Uncertainty】propagated to the conclusion-bearing outputs? [Y/N]
96【Trade-offs】scenarios compared on multiple objectives (Pareto/curve)
97【Reproducible】master workflow + pinned model/version + seeds? [Y/N]
98【Next】agsy-figures-and-tables
99```
100
101## Supplementary resources
102
103- [`../../resources/external_tools.md`](../../resources/external_tools.md) — calibration, sensitivity, uncertainty, and multi-objective packages
104- [`../../resources/official-source-map.md`](../../resources/official-source-map.md) — research-data/model reproducibility policy
105
106---
107
108**Source:** [`brycewang-stanford/Awesome-Journal-Skills`](https://github.com/brycewang-stanford/Awesome-Journal-Skills) → `Agricultural-Systems-Skills/skills/agsy-data-and-model-evaluation/SKILL.md`
thedixitjain