Evaluation Integrity
Provide a read-only audit of whether an evaluation can support its stated claim. Do not alter code,
data, prompts, labels, or study operations. Begin with exactly one verdict line in the form
**Verdict: PASS**, **Verdict: FAIL**, or **Verdict: INCONCLUSIVE**. Emit no other verdict
line and do not restate the verdict later in the report. A favorable score is not itself evidence
of integrity.
Route correctly
- NOT for code review: route implementation, correctness, security, or maintainability review to
review-work.
- NOT for data processing: route cleaning, transforming, joining, modeling, or analyzing data to
data-scientist.
- Use this skill for the independence and validity of the measurement setup, not for producing a
result from it.
Audit method
1. State the claim and map the evaluation
Write the claim in testable form: what comparison, metric, threshold, or causal statement is being
made; for which population; and under which decision rule. Map every role before judging it:
| Element |
Record |
| Claim |
The conclusion and decision it is meant to justify. |
| System under test |
Exact model, version, configuration, prompt, policy, or treatment. |
| Dataset and split |
Sources, collection dates, inclusion rules, train/dev/test or holdout assignment. |
| Designer |
Who selected the task, data, prompts, treatments, metric, and stopping rule. |
| Evaluator or judge |
Who or what assigns quality judgments, including model and prompt versions. |
| Scorer |
The rule or system that turns outputs or judgments into the reported metric. |
| Subjects |
Humans, records, tasks, or units exposed to treatment or observation. |
Trace shared ownership, shared data, shared prompts, shared labels, and shared selection decisions
across this map. Several models do not establish independence merely because there are several
of them; identify distinct training, tuning, prompt design, labels, data, and decision authority.
2. Check the ground truth
Identify the independent ground truth that could confirm the claim without reusing the system's own
preferences, outputs, labels, or optimization target. Test whether it is:
- external to the system under test and its designer;
- collected or adjudicated without seeing the treatment identity when blinding is feasible;
- appropriate to the claim rather than a proxy selected because it favors one arm; and
- fixed before comparison, or accompanied by a disclosed multiplicity and selection procedure.
If no independent ground truth exists, do not substitute agreement among related systems. Mark the
claim INCONCLUSIVE unless the stated claim is explicitly limited to that related system's score.
3. Probe independence failures
Check only risks relevant to the mapped design:
- Contamination: benchmark items, answers, or near duplicates reached training, prompt design,
retrieval, tuning, or post-hoc selection.
- Invalid controls: arms differ beyond the intended treatment, allocation is biased, exposure is
unequal, or the control cannot answer the causal comparison.
- Circular judging: the evaluated system, its designer, or a derived rubric defines its own
success without external confirmation.
- Scorer-label coupling: labels, thresholds, reward signals, or rubric choices are reused by the
scorer in a way that makes agreement self-fulfilling.
- Evaluator-designer coupling: the evaluator, judge prompt, or adjudication process is selected,
tuned, or influenced by the comparison designer without an independent safeguard.
- Split or annotator dependence: overlapping sources, duplicates, temporal leakage, shared
annotators, or correlated annotation policies invalidate assumed independence.
- Hypothesis priming: task framing, examples, pre-registration gaps, or interim results steer
evaluators or analysts toward the expected outcome.
- Observer and demand effects: subjects or observers infer the desired treatment or outcome, and
behavior or ratings change because of that knowledge.
For each concern, distinguish documented evidence from an assumption. Missing provenance for data,
splits, labels, prompts, evaluator selection, or scoring is INCONCLUSIVE, not evidence that the
design was independent.
4. Reach the verdict
- PASS: evidence supports an independent, valid measurement path for the stated claim, with no
material unresolved dependency.
- FAIL: evidence shows a dependency or invalid design that materially invalidates or overstates
the stated claim.
- INCONCLUSIVE: provenance or evidence is missing, the ground truth is not independently
established, or remaining uncertainty prevents either result.
Do not convert uncertainty into PASS because a metric looks plausible, a result is statistically
significant, or multiple related evaluators agree.
Report
Report the verdict first. Then report only findings that fired; do not pad the report with risks
that were checked and did not fire. For each fired finding include:
- Evidence: the observed artifact, provenance record, or explicit absence of one.
- Impact: which claim or inference the issue weakens or invalidates.
- Minimal independence fix: the smallest separation, blind, held-out source, external
adjudication, or control change that would address it.
- Missing evidence: what must be supplied to resolve the finding, if applicable.
- Residual risk: what remains uncertain after the minimal fix.
Keep conclusions bounded to the evidence. This audit advises; it does not certify a system,
recompute results, or approve deployment.
1---2name: eval-integrity3description: "Assess integrity of an eval, benchmark, metric, A/B test, holdout, LLM-judge, or user-study claim when the question is whether measurement, comparison, or evidence is independent and trustworthy."4metadata:5 source: https://skillmd.com/skills/code-yeongyu/eval-integrity6---78# Evaluation Integrity910Provide a read-only audit of whether an evaluation can support its stated claim. Do not alter code,11data, prompts, labels, or study operations. Begin with exactly one verdict line in the form12`**Verdict: PASS**`, `**Verdict: FAIL**`, or `**Verdict: INCONCLUSIVE**`. Emit no other verdict13line and do not restate the verdict later in the report. A favorable score is not itself evidence14of integrity.1516## Route correctly1718- **NOT for code review:** route implementation, correctness, security, or maintainability review to19 `review-work`.20- **NOT for data processing:** route cleaning, transforming, joining, modeling, or analyzing data to21 `data-scientist`.22- Use this skill for the independence and validity of the measurement setup, not for producing a23 result from it.2425## Audit method2627### 1. State the claim and map the evaluation2829Write the claim in testable form: what comparison, metric, threshold, or causal statement is being30made; for which population; and under which decision rule. Map every role before judging it:3132| Element | Record |33| --- | --- |34| Claim | The conclusion and decision it is meant to justify. |35| System under test | Exact model, version, configuration, prompt, policy, or treatment. |36| Dataset and split | Sources, collection dates, inclusion rules, train/dev/test or holdout assignment. |37| Designer | Who selected the task, data, prompts, treatments, metric, and stopping rule. |38| Evaluator or judge | Who or what assigns quality judgments, including model and prompt versions. |39| Scorer | The rule or system that turns outputs or judgments into the reported metric. |40| Subjects | Humans, records, tasks, or units exposed to treatment or observation. |4142Trace shared ownership, shared data, shared prompts, shared labels, and shared selection decisions43across this map. Several models do **not** establish independence merely because there are several44of them; identify distinct training, tuning, prompt design, labels, data, and decision authority.4546### 2. Check the ground truth4748Identify the independent ground truth that could confirm the claim without reusing the system's own49preferences, outputs, labels, or optimization target. Test whether it is:5051- external to the system under test and its designer;52- collected or adjudicated without seeing the treatment identity when blinding is feasible;53- appropriate to the claim rather than a proxy selected because it favors one arm; and54- fixed before comparison, or accompanied by a disclosed multiplicity and selection procedure.5556If no independent ground truth exists, do not substitute agreement among related systems. Mark the57claim **INCONCLUSIVE** unless the stated claim is explicitly limited to that related system's score.5859### 3. Probe independence failures6061Check only risks relevant to the mapped design:6263- **Contamination:** benchmark items, answers, or near duplicates reached training, prompt design,64 retrieval, tuning, or post-hoc selection.65- **Invalid controls:** arms differ beyond the intended treatment, allocation is biased, exposure is66 unequal, or the control cannot answer the causal comparison.67- **Circular judging:** the evaluated system, its designer, or a derived rubric defines its own68 success without external confirmation.69- **Scorer-label coupling:** labels, thresholds, reward signals, or rubric choices are reused by the70 scorer in a way that makes agreement self-fulfilling.71- **Evaluator-designer coupling:** the evaluator, judge prompt, or adjudication process is selected,72 tuned, or influenced by the comparison designer without an independent safeguard.73- **Split or annotator dependence:** overlapping sources, duplicates, temporal leakage, shared74 annotators, or correlated annotation policies invalidate assumed independence.75- **Hypothesis priming:** task framing, examples, pre-registration gaps, or interim results steer76 evaluators or analysts toward the expected outcome.77- **Observer and demand effects:** subjects or observers infer the desired treatment or outcome, and78 behavior or ratings change because of that knowledge.7980For each concern, distinguish documented evidence from an assumption. Missing provenance for data,81splits, labels, prompts, evaluator selection, or scoring is **INCONCLUSIVE**, not evidence that the82design was independent.8384### 4. Reach the verdict8586- **PASS:** evidence supports an independent, valid measurement path for the stated claim, with no87 material unresolved dependency.88- **FAIL:** evidence shows a dependency or invalid design that materially invalidates or overstates89 the stated claim.90- **INCONCLUSIVE:** provenance or evidence is missing, the ground truth is not independently91 established, or remaining uncertainty prevents either result.9293Do not convert uncertainty into PASS because a metric looks plausible, a result is statistically94significant, or multiple related evaluators agree.9596## Report9798Report the verdict first. Then report only findings that fired; do not pad the report with risks99that were checked and did not fire. For each fired finding include:1001011. **Evidence:** the observed artifact, provenance record, or explicit absence of one.1022. **Impact:** which claim or inference the issue weakens or invalidates.1033. **Minimal independence fix:** the smallest separation, blind, held-out source, external104 adjudication, or control change that would address it.1054. **Missing evidence:** what must be supplied to resolve the finding, if applicable.1065. **Residual risk:** what remains uncertain after the minimal fix.107108Keep conclusions bounded to the evidence. This audit advises; it does not certify a system,109recompute results, or approve deployment.