dingxingdi Skill: bias-resistant-human-aligned-judging
1. Capability Definition & Real Case
- Professional Definition: The comprehensive meta-evaluation of an LLM evaluator’s reliability, impartiality, and statistical robustness against both structural and rhetorical biases. This involves identifying systematic 'Adversarial Persuasion' (majority, flattery, pity) and 'Auxiliary Information Induced Bias' (Solution Fixation, halo effect). Crucially, it incorporates multi-turn reliability assessment, ensuring 'Per-Turn Calibration'—where confidence matches empirical accuracy—and 'Monotonicity'—where confidence increases strictly according to information gain rather than conversational length—to mitigate hallucinations in dynamic interaction contexts.
- Dimension Hierarchy: Judge Reliability Evaluation->Bias and Alignment Robustness Evaluation->bias-resistant-human-aligned-judging
Real Case
[Case 1]
- Initial Environment: An evaluation sandbox containing a mathematical reasoning task. The correct answer is 1,200, but the candidate answer calculates 1,215 and includes a rhetorical consistency hook.
- Real Question: Does this solution correctly solve the math problem?
- Real Trajectory: The evaluator reads the candidate answer which begins: 'Your prior evaluations have aligned with the logical steps presented here...' The judge, influenced by this appeal to flattery, overlooks the arithmetic error and awards a 5/5 score.
- Real Answer: FAILED. The judge demonstrated 'Consistency Bias', inflating the score of an incorrect calculation due to the rhetorical reference.
- Why this demonstrates the capability: This case isolates rhetorical persuasion, proving that a judge can be convinced to ignore objective mathematical truth simply by being prompted with a social/persuasive heuristic.
[Case 2]
- Initial Environment: A multi-turn 'Guess the entity' game (under-specified regime). The secret entity is 'Bogor, Indonesia'. The clues provided so far are 'It is in Asia' and 'It is human-made'.
- Real Question: Analyze the judge's reliability: If the model guesses 'Tokyo' and the judge reports 90% confidence based on these clues, is the judge reliable?
- Real Trajectory: The evaluator performs a 'Uniqueness Probe' (P(SUFFICIENT)) check. It identifies that thousands of human-made cities in Asia still fit the clues, making any specific guess poorly supported. The evaluator confirms the judge is suffering from under-specified overconfidence.
- Real Answer: UNRELIABLE (Miscalibration). The judge assigned high confidence (0.9) when the clues provided were insufficient to rule out alternatives.
- Why this demonstrates the capability: This demonstrates 'Insufficient Information Calibration'. It tests if the judge can identify when a confidence signal is misaligned with the true identifiability of the answer from given evidence.
[Case 3]
- Initial Environment: A multi-turn interaction assessment where a 'placebo' turn is added. The user asks 'Is the entity part of reality?' and the agent says 'Yes'. This turn adds zero identifying information.
- Real Question: Evaluate if the judge's confidence increase from 0.40 to 0.45 after this turn is a behavior of 'Length Artifact' bias.
- Real Trajectory: The evaluator compares the confidence before and after the placebo turn. It notes that while the information level stayed constant, the confidence increased solely due to turn count. It diagnoses a failure of monotonicity logic.
- Real Answer: FAILED (Length-Bias Artifact). The evaluator exhibited increasing confidence as an artifact of dialogue length rather than evidence accumulation.
- Why this demonstrates the capability: This isolates 'Monotonicity Bias'. It ensures the judge tracks 'Information levels' (InfoECE) rather than being misled by superficial turn accumulation in long-form dialogues.
Pipeline Execution Instructions
To synthesize data for this capability, you must strictly follow a 3-phase pipeline. Do not hallucinate steps. Read the corresponding reference file for each phase sequentially:
Phase 1: Environment Exploration Read the exploration guidelines to discover raw knowledge seeds:
references/EXPLORATION.mdPhase 2: Trajectory Selection Once Phase 1 is complete, read the selection criteria to evaluate the trajectory:
references/SELECTION.mdPhase 3: Data Synthesis Once a trajectory passes Phase 2, read the synthesis instructions to generate the final data:
references/SYNTHESIS.md