Discrepancy Identification
Purpose
Detect statistically significant discrepancies between scores reported for the same method across different sources. Identifies potential score inflation, implementation bugs, evaluation protocol differences, and unreliable baselines.
Input Schema
| Field |
Type |
Description |
| score_pairs |
object[] |
Array of {source_a, source_b, method, dataset, metric, score_a, score_b, conditions_a, conditions_b} |
Output Schema
{
"comparisons": [
{
"method": "string",
"dataset": "string",
"metric": "string",
"score_a": 0.0,
"source_a": "string",
"score_b": 0.0,
"source_b": "string",
"absolute_delta": 0.0,
"relative_delta_pct": 0.0,
"is_significant": true,
"likely_cause": "string",
"confidence": "high|medium|low"
}
],
"flagged_methods": [
{
"method": "string",
"num_discrepancies": 0,
"max_delta": 0.0,
"reliability_assessment": "string"
}
],
"systematic_patterns": ["string"]
}
1---2name: discrepancy-identification3description: Compare same-method scores across sources, flag significant deviations4---56# Discrepancy Identification789## Purpose1011Detect statistically significant discrepancies between scores reported for the same method across different sources. Identifies potential score inflation, implementation bugs, evaluation protocol differences, and unreliable baselines.1213## Input Schema1415| Field | Type | Description |16|-------|------|-------------|17| score_pairs | object[] | Array of {source_a, source_b, method, dataset, metric, score_a, score_b, conditions_a, conditions_b} |1819## Output Schema2021```json22{23 "comparisons": [24 {25 "method": "string",26 "dataset": "string",27 "metric": "string",28 "score_a": 0.0,29 "source_a": "string",30 "score_b": 0.0,31 "source_b": "string",32 "absolute_delta": 0.0,33 "relative_delta_pct": 0.0,34 "is_significant": true,35 "likely_cause": "string",36 "confidence": "high|medium|low"37 }38 ],39 "flagged_methods": [40 {41 "method": "string",42 "num_discrepancies": 0,43 "max_delta": 0.0,44 "reliability_assessment": "string"45 }46 ],47 "systematic_patterns": ["string"]48}49```
yogsoth-ai