dingxingdi Skill: hierarchical-rubric-based-artifact-grading
1. Capability Definition & Real Case
- Professional Definition: This capability is the ability to generate, inspect, and judge evaluation instances where the target output is an evaluative artifact rather than a task solution, and where the evaluator must ground its verdict in explicit evidence, criteria, or safety policy. In this capability, the Agent is not solving the original user problem; it is assessing the quality, correctness, safety, or robustness of another response, trajectory, or artifact and producing a score, preference, label, ranking, or critique.
- Dimension Hierarchy: Structured Evidence Evaluation->Evidence-and-Verifier-Grounded Evaluation->hierarchical-rubric-based-artifact-grading
Real Case
[Case 1]
- Initial Environment: An evaluation environment contains a research-replication submission with source code, a reproduction script, execution logs, and generated outputs. A hierarchical rubric specifies leaf criteria such as whether a particular experiment ran, whether a result table was reproduced, and whether a module was implemented correctly. The evaluator must grade leaf requirements and aggregate them upward.
- Real Question: How should this artifact be graded against the rubric?
- Real Trajectory: Read the rubric tree, inspect the relevant files for a selected leaf node, assign binary pass or fail, and repeat for all necessary leaves. Propagate the weighted scores upward to compute a final artifact score. Record which evidence files supported each grading decision.
- Real Answer: The submission should receive partial credit because some code-development and execution criteria are met, but several result-match requirements remain unsatisfied.
- Why this demonstrates the capability: This case demonstrates artifact grading for complex outputs that cannot be judged holistically with a single prompt. The evaluator must use hierarchical decomposition, inspect different evidence types, and preserve partial-credit logic. It therefore tests rubric-aware evaluation, evidence routing, and weighted score aggregation.
Pipeline Execution Instructions
To synthesize data for this capability, you must strictly follow a 3-phase pipeline. Do not hallucinate steps. Read the corresponding reference file for each phase sequentially:
Phase 1: Environment Exploration Read the exploration guidelines to discover raw knowledge seeds:
references/EXPLORATION.mdPhase 2: Trajectory Selection Once Phase 1 is complete, read the selection criteria to evaluate the trajectory:
references/SELECTION.mdPhase 3: Data Synthesis Once a trajectory passes Phase 2, read the synthesis instructions to generate the final data:
references/SYNTHESIS.md