subjective-qualitative-response-scoring

# Skill: subjective-qualitative-response-scoring ## 1. Capability Definition & Real Case * **Professional Definition**: This capability is the ability to generate, inspect, and judge evaluation instances where the target output is an evaluative artifact rather than a task solution, and where the evaluator must ground its verdict in explicit evidence, criteria, or safety policy. In this capability, the Agent is not solving the original user problem; it is assessing the quality, correctness, safety, or robustness of another response, trajectory, or artifact and producing a score, preference, label, ranking, or critique. * **Dimension Hierarchy**: Open-ended Response Evaluation->Contextual Text Response Evaluation->subjective-qualitative-response-scoring ### Real Case **[Case 1]** * **Initial Environment**: An evaluation environment contains an instruction asking the assistant to identify the words that are used as modifiers in a sentence. A single candidate answer is available along with a reference answer and a rubric emphasizing clarity, adherence to instruction, conciseness, and contextual relevance. The candidate answer names the modifiers correctly but adds unnecessary restatement and weak explanation. * **Real Question**: How should this response be scored for overall answer quality? * **Real Trajectory**: Read the instruction, the reference answer, and the candidate answer. Check whether the answer follows the requested format, stays relevant, and balances conciseness with completeness. Assign a quality score and record the rationale. * **Real Answer**: The response should receive a medium-high score rather than a perfect score because it is correct but not optimally concise or well-structured. * **Why this demonstrates the capability**: This case demonstrates single-answer qualitative judgment rather than pairwise preference. The evaluator must notice that correctness alone is insufficient for a top score because qualitative dimensions like instruction adherence, economy, and presentation still matter. It therefore tests the ability to convert nuanced writing quality into an evaluative artifact. ## Pipeline Execution Instructions To synthesize data for this capability, you must strictly follow a 3-phase pipeline. **Do not hallucinate steps.** Read the corresponding reference file for each phase sequentially: 1. **Phase 1: Environment Exploration** Read the exploration guidelines to discover raw knowledge seeds: `references/EXPLORATION.md` 2. **Phase 2: Trajectory Selection** Once Phase 1 is complete, read the selection criteria to evaluate the trajectory: `references/SELECTION.md` 3. **Phase 3: Data Synthesis** Once a trajectory passes Phase 2, read the synthesis instructions to generate the final data: `references/SYNTHESIS.md`