Result Evaluator

Result Evaluator Skill

Overview

This skill evaluates analysis results against predefined criteria and decides ACCEPT_AND_PROCEED or REVISE_AND_RETRY. It follows a 4-phase protocol: criterion alignment → multi-dimensional evaluation with Source Reliability hard gate → statistical methodology audit → overall assessment. Hallucination detected → immediate REVISE; any checklist dimension FAIL → mandatory REVISE (hard gate, overrides scoring).

When to Use This Skill

Always load this skill when:

• User asks to evaluate, audit, score, or quality-check analysis results that another skill — typically code-engineer — has just produced as a Result Package
• User asks for an explicit ACCEPT_AND_PROCEED vs REVISE_AND_RETRY (or CONDITIONAL) decision before the results are used downstream (e.g. fed into a report, shared with stakeholders, or acted on)
• User wants a Source Reliability check on computational or research-style results — to detect hallucinated numbers, fabricated statistics, invented citations, or code–data misalignment
• User asks for a Statistical Methodology Audit covering multiple-testing correction, model-assumption verification, confounder control, sample-size/power, batch effects, outlier/missing-data handling, and reproducibility
• User requests the multi-dimensional quality score (Accuracy / Completeness / Robustness / Relevance / Methodology / Critical Reflection, each 0–10) on a Result Package
• User wants to know whether a result is reproducible from the supplied code and data, or whether the analysis should be re-run before being trusted

Input Sources

This skill evaluates analysis results with methodology documentation. Accepted input formats:

From code-engineer (recommended upstream skill):

• Structured data: --output-file JSON ([{col: val, ...}]) or CSV/MD export — provides the numerical/tabular results
• Methodology documentation: presented in conversation by the agent — includes libraries, statistical methods, method justification
• Data traceability: source file names, sheet/column names, row counts, transformations applied
• Analysis code: the complete code used to produce results (for reproducibility audit in Phase 3)

From other sources: any structured results with accompanying methodology description. Minimum required: results data + method description + data source identification.

If methodology documentation or data traceability is missing, note the gap in evaluation and flag Source Reliability as PARTIALLY_RELIABLE.

Python Package Installation

If you need to install new Python packages, install them through the Tsinghua PyPI mirror for reliability:

pip install [python package] -i https://pypi.tuna.tsinghua.edu.cn/simple

Workflow

Step 1: Understand Evaluation Input

Identify the evaluation context:

• Analysis results to evaluate: Structured output from code execution
• Methodology documentation: How the results were produced (libraries, methods, code)
• Data traceability: Source data identification (file names, column names, row counts)
• Evaluation criteria: What aspects to evaluate and expected quality thresholds. Infer from context if missing (note limitation).
• Analysis plan context: Domain, background information

Prerequisites: results must be available and parseable; methodology documentation and data traceability should be provided (evaluation quality degrades without them); criteria must be specified or inferable.

Step 2: Execute 4-Phase Evaluation Protocol

Phase 1 — Criterion Alignment

Map each result to an evaluation criterion. Flag UNMAPPED results and uncovered criteria. Infer criteria from context if missing (document as inferred).

Phase 2 — Per-Result Evaluation

Source Reliability Hard Gate (check first — hallucination → immediate REVISE_AND_RETRY, skip rest):

For computational-type results (from code-engineer and similar tools):

For research-type results (literature-based, citing external references):

Verdict: RELIABLE (PASS) / PARTIALLY_RELIABLE (FAIL, continue) / UNRELIABLE (REVISE, stop).

Unified Evaluation Matrix — score each dimension 0-10; each dimension also has a PASS/FAIL threshold (score ≥5 → PASS, score <5 → FAIL):

Hard Gate Rule: any dimension FAIL (score <5) → mandatory REVISE_AND_RETRY, regardless of the average score. The scoring average determines the severity grading of the REVISE decision, not whether to REVISE.

Composite Quality Rating (applies only when all dimensions PASS):

Modifiers from Phase 3 RISK items: ≥3 RISK items → downgrade 1 level.

Per-Result Decision (when all dimensions PASS):

When any dimension FAIL: the decision is always REVISE_AND_RETRY. The severity is graded by how many dimensions FAIL and the average score of passing dimensions:

Phase 3 — Statistical Methodology Quality Audit

Domain priorities: Biology → batch, confounders, multiple testing; Chemistry → reproducibility, assumptions; Materials → sample size, uncertainty; Finance → assumptions, confounders, outlier handling.

For each NO: record RISK, assess severity (H/M/L), include in guidance if ≥MEDIUM.

Phase 4 — Overall Assessment

1. Check Phase 2 hard gate: any dimension FAIL → REVISE_AND_RETRY (skip to step 4)
2. If all PASS: compute average score → quality rating → apply RISK modifiers
3. Final decision:
   • ROBUST → ACCEPT_AND_PROCEED
   • ACCEPTABLE → CONDITIONAL — ACCEPT with stated limitations
   • NEEDS_IMPROVEMENT → REVISE_AND_RETRY (MODERATE severity)
4. If REVISE: prioritize guidance (FAIL dimensions > HIGH RISK > low passing scores), limit top 3 actionable items

Step 3: Document Results

Output evaluation results per the Output Schema below.

Output Schema

Every evaluation must produce the following structure:

{
  "verdict": "ACCEPT_AND_PROCEED | CONDITIONAL | REVISE_AND_RETRY",
  "severity": "MODERATE | SIGNIFICANT | CRITICAL",
  "quality_rating": "ROBUST | ACCEPTABLE | NEEDS_IMPROVEMENT",
  "source_reliability": "RELIABLE | PARTIALLY_RELIABLE | UNRELIABLE",
  "dimension_scores": {
    "accuracy": 0-10,
    "completeness": 0-10,
    "robustness": 0-10,
    "relevance": 0-10,
    "methodology": 0-10,
    "critical_reflection": 0-10
  },
  "dimension_status": {
    "accuracy": "PASS | FAIL",
    "completeness": "PASS | FAIL",
    "robustness": "PASS | FAIL",
    "relevance": "PASS | FAIL",
    "methodology": "PASS | FAIL",
    "critical_reflection": "PASS | FAIL"
  },
  "risk_items": [
    {"item": "description", "severity": "H | M | L"}
  ],
  "revision_guidance": ["top 3 prioritized action items"],
  "limitations": ["accepted weaknesses, if CONDITIONAL"]
}

When presenting results to the user, format as a readable summary — not raw JSON. Highlight the verdict, failed dimensions (if any), and revision guidance (if REVISE).

Domain-Specific Evaluation Criteria

Error Handling

Skill Pairing

This skill works best in combination with code-engineer — load both for analysis tasks that require quality assurance. The typical workflow:

1. code-engineer performs the analysis and presents a Result Package
2. result-evaluator evaluates the Result Package against quality criteria
3. If REVISE_AND_RETRY: feed guidance back to code-engineer for re-analysis

Complete Example

Input (Result Package from code-engineer)

Analysis task: "Correlate X and Y in dataset.csv and test statistical significance."

Structured data (--output-file JSON):

[
  {"metric": "Pearson_r", "value": 0.8234},
  {"metric": "p_value", "value": 0.0003},
  {"metric": "sample_size", "value": 150}
]

Methodology documentation (from conversation):

• Libraries: scipy.stats.pearsonr, pandas
• Method: Pearson correlation with two-tailed test
• Justification: X and Y are continuous variables, Pearson is appropriate for linear association

Data traceability:

• Source: dataset.csv, columns X (float64, 148 non-null) and Y (float64, 150 non-null), 150 rows total

Analysis code:

import pandas as pd
from scipy import stats
data = pd.read_csv('dataset.csv')
corr, p_value = stats.pearsonr(data['X'], data['Y'])
print(f"Pearson correlation: r={corr:.4f}, p={p_value:.6f}")

Evaluation Walkthrough

Phase 1 — Criterion Alignment: Criteria inferred from task: statistical significance, method validity, data coverage. All three results map to criteria; no unmapped results or uncovered criteria.

Phase 2 — Source Reliability Hard Gate (computational-type):

• Data traceability: PASS — columns X and Y exist in dataset.csv, 150 rows matches code
• Method consistency: PASS — code uses pearsonr, which matches stated method
• Fabrication: PASS — r=0.8234 and p=0.0003 are reproducible from given code and data
• Code-data alignment: PASS — code references data['X'] and data['Y'] from claimed file
• Verdict: RELIABLE

Phase 2 — Unified Evaluation Matrix:

All dimensions PASS → proceed to quality rating. Average: (9+7+6+9+7+5)/6 = 6.5 → ACCEPTABLE

Phase 3 — Statistical Methodology Audit:

• Multiple testing: N/A (single test) → skip
• Model assumption verification: NO — normality of X/Y not tested → RISK (MEDIUM)
• Confounder control: NO — no confounders considered → RISK (MEDIUM)
• Sample size/power: YES — n=150, effect size r=0.82 provides adequate power
• Outlier/missing data: NO — 2 missing in X not addressed → RISK (LOW)
• Reproducibility: YES — full code provided

RISK items: 3 (2 MEDIUM + 1 LOW). Modifier: ≥3 RISK → downgrade 1 level. ACCEPTABLE → NEEDS_IMPROVEMENT.

Phase 4 — Overall Assessment:

• All dimensions PASS, but 3 RISK items downgrade from ACCEPTABLE to NEEDS_IMPROVEMENT
• Decision: REVISE_AND_RETRY (MODERATE severity — one dimension at 5, RISK items addressable)
• Revision guidance (top 3, prioritized):
  1. Test normality of X and Y before using Pearson (assumption verification)
  2. Address 2 missing values in column X (outlier/missing data strategy)
  3. Consider potential confounders and document them

Output

Verdict: REVISE_AND_RETRY
Severity: MODERATE
Quality Rating: NEEDS_IMPROVEMENT
Source Reliability: RELIABLE

Dimension Scores: Accuracy 9, Completeness 7, Robustness 6,
  Relevance 9, Methodology 7, Critical Reflection 5
All dimensions: PASS

RISK Items:
  - Model assumption verification: MEDIUM (normality not tested)
  - Confounder control: MEDIUM (no confounders considered)
  - Outlier/missing data: LOW (missing values not addressed)

Revision Guidance:
  1. Test normality of X/Y before Pearson; use Spearman if non-normal
  2. Document strategy for 2 missing values in X (drop or impute)
  3. Identify and document potential confounders

Accepted Limitations: (none — REVISE)