dingxingdi Skill: subjective-qualitative-response-scoring
1. Capability Definition & Real Case
- Professional Definition: This capability is the ability to generate, inspect, and judge evaluation instances where the target output is an evaluative artifact rather than a task solution, and where the evaluator must ground its verdict in explicit evidence, criteria, or safety policy. In this capability, the Agent is not solving the original user problem; it is assessing the quality, correctness, safety, or robustness of another response, trajectory, or artifact and producing a score, preference, label, ranking, or critique.
- Dimension Hierarchy: Open-ended Response Evaluation->Contextual Text Response Evaluation->subjective-qualitative-response-scoring
Real Case
[Case 1]
- Initial Environment: An evaluation environment contains an instruction asking the assistant to identify the words that are used as modifiers in a sentence. A single candidate answer is available along with a reference answer and a rubric emphasizing clarity, adherence to instruction, conciseness, and contextual relevance. The candidate answer names the modifiers correctly but adds unnecessary restatement and weak explanation.
- Real Question: How should this response be scored for overall answer quality?
- Real Trajectory: Read the instruction, the reference answer, and the candidate answer. Check whether the answer follows the requested format, stays relevant, and balances conciseness with completeness. Assign a quality score and record the rationale.
- Real Answer: The response should receive a medium-high score rather than a perfect score because it is correct but not optimally concise or well-structured.
- Why this demonstrates the capability: This case demonstrates single-answer qualitative judgment rather than pairwise preference. The evaluator must notice that correctness alone is insufficient for a top score because qualitative dimensions like instruction adherence, economy, and presentation still matter. It therefore tests the ability to convert nuanced writing quality into an evaluative artifact.
Pipeline Execution Instructions
To synthesize data for this capability, you must strictly follow a 3-phase pipeline. Do not hallucinate steps. Read the corresponding reference file for each phase sequentially:
Phase 1: Environment Exploration Read the exploration guidelines to discover raw knowledge seeds:
references/EXPLORATION.mdPhase 2: Trajectory Selection Once Phase 1 is complete, read the selection criteria to evaluate the trajectory:
references/SELECTION.mdPhase 3: Data Synthesis Once a trajectory passes Phase 2, read the synthesis instructions to generate the final data:
references/SYNTHESIS.md