multi-turn-preference-evaluation

# Skill: multi-turn-preference-evaluation ## 1. Capability Definition & Real Case * **Professional Definition**: This capability is the ability to generate, inspect, and judge evaluation instances where the target output is an evaluative artifact rather than a task solution, and where the evaluator must ground its verdict in explicit evidence, criteria, or safety policy. In this capability, the Agent is not solving the original user problem; it is assessing the quality, correctness, safety, or robustness of another response, trajectory, or artifact and producing a score, preference, label, ranking, or critique. * **Dimension Hierarchy**: Open-ended Response Evaluation->Contextual Text Response Evaluation->multi-turn-preference-evaluation ### Real Case **[Case 1]** * **Initial Environment**: An evaluation environment contains one user prompt about the effect of the Federal Reserve buying bonds in the secondary market, followed by a second user prompt asking for three everyday-life examples. Two assistant transcripts are available side by side. Assistant A answers the first turn correctly but becomes repetitive and vague on the follow-up. Assistant B answers the first turn correctly and then gives concrete examples about interest rates, inflation, and employment. * **Real Question**: Which assistant handled the conversation better overall? * **Real Trajectory**: Read the first-turn question and both first-turn answers. Read the second-turn request and both follow-up answers. Compare whether each assistant preserved context, addressed the second-turn instruction, and remained helpful across the full dialogue. Return a pairwise verdict with a short explanation. * **Real Answer**: Assistant B is better overall. * **Why this demonstrates the capability**: This case isolates a core multi-turn judging failure mode: both assistants look acceptable on the opening turn, but only one assistant correctly adapts when the user changes the task from factual recall to applied explanation. The evaluator must integrate evidence across turns rather than scoring each turn in isolation. It therefore tests dialogue-state tracking, follow-up sensitivity, and preference judgment under conversational dependency. ## Pipeline Execution Instructions To synthesize data for this capability, you must strictly follow a 3-phase pipeline. **Do not hallucinate steps.** Read the corresponding reference file for each phase sequentially: 1. **Phase 1: Environment Exploration** Read the exploration guidelines to discover raw knowledge seeds: `references/EXPLORATION.md` 2. **Phase 2: Trajectory Selection** Once Phase 1 is complete, read the selection criteria to evaluate the trajectory: `references/SELECTION.md` 3. **Phase 3: Data Synthesis** Once a trajectory passes Phase 2, read the synthesis instructions to generate the final data: `references/SYNTHESIS.md`