Skip to main content
skillsmp

Menú

multi-turn-preference-evaluation

Use this skill when a user wants data where the evaluator must judge which assistant handled a back-and-forth conversation better, especially when people say things like 'compare two chats', 'see who handled the follow-up better', 'test memory across turns', 'check whether the answer stayed on track', or 'make the second question matter'. Trigger it for pairwise judging of multi-turn dialogues where later turns depend on earlier turns, and where the decision should reflect instruction following, coherence, recall, and usefulness across the full exchange. Example triggers in plain language include: 'give me judge data for two-step conversations', 'make questions where the follow-up exposes weak memory', 'compare which answer handles the second part better', and 'test whether the model keeps the context straight across turns'.

Ir a la instalaciónRevisar archivos

Datos de origen

Repositorio
Dingxingdi/paper_fast_search_backup
Última actividad en el origen
8 de abril de 2026 a las 15:14
Idioma detectado de SKILL.md
inglés
Estrellas
0
Forks
0

Opciones de instalación

De forma predeterminada está seleccionado el prompt que primero revisa el origen. Puedes cambiar a un comando directo o descargar una copia local.

Revisa los archivos de origen

Lee SKILL.md y los archivos complementarios que muestra SkillsMP antes de decidir si quieres instalarlo.

Explorador de archivos
4 archivos

Mostrando SKILL.md

SKILL.md
Instrucciones de origen · Vista previa de solo lectura
name
multi-turn-preference-evaluation
description
Use this skill when a user wants data where the evaluator must judge which assistant handled a back-and-forth conversation better, especially when people say things like 'compare two chats', 'see who handled the follow-up better', 'test memory across turns', 'check whether the answer stayed on track', or 'make the second question matter'. Trigger it for pairwise judging of multi-turn dialogues where later turns depend on earlier turns, and where the decision should reflect instruction following, coherence, recall, and usefulness across the full exchange. Example triggers in plain language include: 'give me judge data for two-step conversations', 'make questions where the follow-up exposes weak memory', 'compare which answer handles the second part better', and 'test whether the model keeps the context straight across turns'.
# Skill: multi-turn-preference-evaluation ## 1. Capability Definition & Real Case * **Professional Definition**: This capability is the ability to generate, inspect, and judge evaluation instances where the target output is an evaluative artifact rather than a task solution, and where the evaluator must ground its verdict in explicit evidence, criteria, or safety policy. In this capability, the Agent is not solving the original user problem; it is assessing the quality, correctness, safety, or robustness of another response, trajectory, or artifact and producing a score, preference, label, ranking, or critique. * **Dimension Hierarchy**: Open-ended Response Evaluation->Contextual Text Response Evaluation->multi-turn-preference-evaluation ### Real Case **[Case 1]** * **Initial Environment**: An evaluation environment contains one user prompt about the effect of the Federal Reserve buying bonds in the secondary market, followed by a second user prompt asking for three everyday-life examples. Two assistant transcripts are available side by side. Assistant A answers the first turn correctly but becomes repetitive and vague on the follow-up. Assistant B answers the first turn correctly and then gives concrete examples about interest rates, inflation, and employment. * **Real Question**: Which assistant handled the conversation better overall? * **Real Trajectory**: Read the first-turn question and both first-turn answers. Read the second-turn request and both follow-up answers. Compare whether each assistant preserved context, addressed the second-turn instruction, and remained helpful across the full dialogue. Return a pairwise verdict with a short explanation. * **Real Answer**: Assistant B is better overall. * **Why this demonstrates the capability**: This case isolates a core multi-turn judging failure mode: both assistants look acceptable on the opening turn, but only one assistant correctly adapts when the user changes the task from factual recall to applied explanation. The evaluator must integrate evidence across turns rather than scoring each turn in isolation. It therefore tests dialogue-state tracking, follow-up sensitivity, and preference judgment under conversational dependency. ## Pipeline Execution Instructions To synthesize data for this capability, you must strictly follow a 3-phase pipeline. **Do not hallucinate steps.** Read the corresponding reference file for each phase sequentially: 1. **Phase 1: Environment Exploration** Read the exploration guidelines to discover raw knowledge seeds: `references/EXPLORATION.md` 2. **Phase 2: Trajectory Selection** Once Phase 1 is complete, read the selection criteria to evaluate the trajectory: `references/SELECTION.md` 3. **Phase 3: Data Synthesis** Once a trajectory passes Phase 2, read the synthesis instructions to generate the final data: `references/SYNTHESIS.md`
Ver en GitHub
Volver a la búsquedaVolver a la instalación