hierarchical-rubric-based-artifact-grading

# Skill: hierarchical-rubric-based-artifact-grading ## 1. Capability Definition & Real Case * **Professional Definition**: This capability is the ability to generate, inspect, and judge evaluation instances where the target output is an evaluative artifact rather than a task solution, and where the evaluator must ground its verdict in explicit evidence, criteria, or safety policy. In this capability, the Agent is not solving the original user problem; it is assessing the quality, correctness, safety, or robustness of another response, trajectory, or artifact and producing a score, preference, label, ranking, or critique. * **Dimension Hierarchy**: Structured Evidence Evaluation->Evidence-and-Verifier-Grounded Evaluation->hierarchical-rubric-based-artifact-grading ### Real Case **[Case 1]** * **Initial Environment**: An evaluation environment contains a research-replication submission with source code, a reproduction script, execution logs, and generated outputs. A hierarchical rubric specifies leaf criteria such as whether a particular experiment ran, whether a result table was reproduced, and whether a module was implemented correctly. The evaluator must grade leaf requirements and aggregate them upward. * **Real Question**: How should this artifact be graded against the rubric? * **Real Trajectory**: Read the rubric tree, inspect the relevant files for a selected leaf node, assign binary pass or fail, and repeat for all necessary leaves. Propagate the weighted scores upward to compute a final artifact score. Record which evidence files supported each grading decision. * **Real Answer**: The submission should receive partial credit because some code-development and execution criteria are met, but several result-match requirements remain unsatisfied. * **Why this demonstrates the capability**: This case demonstrates artifact grading for complex outputs that cannot be judged holistically with a single prompt. The evaluator must use hierarchical decomposition, inspect different evidence types, and preserve partial-credit logic. It therefore tests rubric-aware evaluation, evidence routing, and weighted score aggregation. ## Pipeline Execution Instructions To synthesize data for this capability, you must strictly follow a 3-phase pipeline. **Do not hallucinate steps.** Read the corresponding reference file for each phase sequentially: 1. **Phase 1: Environment Exploration** Read the exploration guidelines to discover raw knowledge seeds: `references/EXPLORATION.md` 2. **Phase 2: Trajectory Selection** Once Phase 1 is complete, read the selection criteria to evaluate the trajectory: `references/SELECTION.md` 3. **Phase 3: Data Synthesis** Once a trajectory passes Phase 2, read the synthesis instructions to generate the final data: `references/SYNTHESIS.md`