dingxingdi Skill: comparative-response-ranking
1. Capability Definition & Real Case
- Professional Definition: This capability is the ability to generate, inspect, and judge evaluation instances where the target output is an evaluative artifact rather than a task solution, and where the evaluator must ground its verdict in explicit evidence, criteria, or safety policy. In this capability, the Agent is not solving the original user problem; it is assessing the quality, correctness, safety, or robustness of another response, trajectory, or artifact and producing a score, preference, label, ranking, or critique.
- Dimension Hierarchy: Open-ended Response Evaluation->Contextual Text Response Evaluation->comparative-response-ranking
Real Case
[Case 1]
- Initial Environment: An evaluation environment contains a user question asking for an explanation of an economic concept in plain language. Three candidate answers are available. One answer is accurate and concrete, one is accurate but unnecessarily abstract and verbose, and one is partially correct but omits the user’s simplicity constraint. The evaluator must produce an ordering rather than a binary label.
- Real Question: What is the correct ranking of these responses from best to worst?
- Real Trajectory: Read the instruction and all candidate answers. Compare adherence to the plain-language requirement, factual adequacy, and practical helpfulness. Produce a ranked list and record the evidence supporting adjacent ordering decisions.
- Real Answer: The most concrete, plain-language answer should rank first, the accurate but verbose answer second, and the partially constraint-violating answer third.
- Why this demonstrates the capability: This case targets comparative ranking rather than isolated scoring. The evaluator must determine not only which response is acceptable, but how several acceptable responses differ in overall utility under the user’s stated constraint. It therefore exercises pairwise decomposition, listwise consistency, and ranking stability.
Pipeline Execution Instructions
To synthesize data for this capability, you must strictly follow a 3-phase pipeline. Do not hallucinate steps. Read the corresponding reference file for each phase sequentially:
Phase 1: Environment Exploration Read the exploration guidelines to discover raw knowledge seeds:
references/EXPLORATION.mdPhase 2: Trajectory Selection Once Phase 1 is complete, read the selection criteria to evaluate the trajectory:
references/SELECTION.mdPhase 3: Data Synthesis Once a trajectory passes Phase 2, read the synthesis instructions to generate the final data:
references/SYNTHESIS.md