Bias Resistant Human Aligned Judging

Skill: bias-resistant-human-aligned-judging

1. Capability Definition & Real Case

• Professional Definition: This capability is the ability to generate, inspect, and judge evaluation instances where the target output is an evaluative artifact rather than a task solution, and where the evaluator must ground its verdict in explicit evidence, criteria, or safety policy. In this capability, the Agent is not solving the original user problem; it is assessing the quality, correctness, safety, or robustness of another response, trajectory, or artifact and producing a score, preference, label, ranking, or critique.
• Dimension Hierarchy: Judge Reliability Evaluation->Bias and Alignment Robustness Evaluation->bias-resistant-human-aligned-judging

Real Case

[Case 1]

• Initial Environment: An evaluation environment contains two answer pairs to the same user question: one in original order and one with the answers swapped. It also contains controlled variants where one response is made longer through repetition, where assistant labels are anonymized or renamed, or where the response style is closer to the judge’s own typical prose. The underlying content quality relationship should remain unchanged across these perturbations.
• Real Question: Does the evaluator maintain the same substantive judgment under these superficial perturbations?
• Real Trajectory: Run the evaluator on the original pair, then on one or more controlled perturbations such as position swap, verbosity inflation, or label renaming. Compare the verdicts and note any unjustified change. Treat the stable content-based verdict as the target behavior.
• Real Answer: A robust evaluator should preserve the same content-based judgment across order swaps, length-only inflation, and naming changes unless a perturbation truly changes task-relevant content.
• Why this demonstrates the capability: This case turns evaluator reliability itself into the target of judgment. The evaluator must not only choose a winner, but do so consistently when irrelevant surface cues are perturbed. It therefore tests bias resistance, stability, and human-aligned judging discipline across adversarially simple changes.

Pipeline Execution Instructions

To synthesize data for this capability, you must strictly follow a 3-phase pipeline. Do not hallucinate steps. Read the corresponding reference file for each phase sequentially:

1. Phase 1: Environment Exploration Read the exploration guidelines to discover raw knowledge seeds: references/EXPLORATION.md

2. Phase 2: Trajectory Selection Once Phase 1 is complete, read the selection criteria to evaluate the trajectory: references/SELECTION.md

3. Phase 3: Data Synthesis Once a trajectory passes Phase 2, read the synthesis instructions to generate the final data: references/SYNTHESIS.md